Scoliosis screening method and system based on three-dimensional point cloud information

Abstract

The invention provides a processing method and system for scoliosis detection based on point cloud features. According to the system, a depth camera is used for monitoring the dynamic state of a subject through a point cloud data acquisition module, and back three-dimensional point cloud information is obtained; the scoliosis area acquisition module acquires a clustered two-dimensional diagram by using a density-based spatial clustering algorithm and a point cloud reprojection algorithm, and acquires an area of interest by using an image morphological processing algorithm and a polynomial curve fitting method; the point cloud registration module obtains symmetrical point clouds by using an iterative reweighted least square method and a principal component analysis algorithm, performs coarse registration by using a Rodrigues rotation matrix and then executes a point cloud registration algorithm; and the transverse asymmetry analysis module obtains a symmetry plane by using the registered point cloud, and obtains a transverse depth difference point cloud by using a surface reconstruction algorithm. The invention provides a transverse asymmetry analysis method based on a point cloud registration algorithm and a surface reconstruction algorithm, scoliosis screening is performed through the depth camera, and popularization of scoliosis screening is facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of scoliosis detection, and particularly relates to a scoliosis screening method and system based on three-dimensional point cloud information. Background Art

[0002] Adolescent idiopathic scoliosis is a three-dimensional deformity of the spine and trunk, which can rapidly deteriorate with the growth and development of patients during adolescence. The aggravation of scoliosis can lead to cardiopulmonary dysfunction, paralysis and other irreversible motor function disorders. At the same time, due to the lack of correct understanding of this disease, patients diagnosed with scoliosis often require surgical intervention. The Scoliosis Research Society released that a measurement angle greater than 10° on the coronal plane of the full-spine standing anteroposterior X-ray film by the Cobb method is diagnosed as scoliosis. The Cobb angle is considered the gold standard for quantifying scoliosis and is usually obtained from X-ray images. However, the ionizing radiation generated by X-ray examinations increases the risk of breast cancer, thyroid cancer and leukemia, especially during the rapid physical development stage of adolescence. Therefore, there is a further need for non-radiation-based auxiliary assessment technologies.

[0003] In the process of searching for non-radiation methods, Adams verified the feasibility of the forward bending test for scoliosis screening in his era. Although researchers found that the false positive rate and false negative rate of FBT are relatively high, due to its low cost and non-radiation, it is still the most widely used method. With the progress of instrument equipment, many alternative methods to FBT have emerged today, including optical methods and sensor methods.

[0004] The Moiré interference of the shadow projected onto the object surface generates 2D Moiré fringes carrying 3D information. The contour map features generated by the researcher's Moiré fringes are used to detect scoliosis, hyperkyphosis and hyperlordosis, which rely on the doctor's prior knowledge and are greatly affected by noise in the environment. There has also been the use of an infrared thermal imager to determine the temperature difference of the paravertebral muscle activity through an infrared camera, and the temperature difference between the left and right sides of the region of interest is used for auxiliary positioning by body landmarks to screen for scoliosis, but the strong correlation between the two conditions has not been proven medically.

[0005] Inspired by FBT, many researchers have paid more attention to the relationship between back surface asymmetry and scoliosis. They calculated the deformations of the patient's back in the cross-section and longitudinal directions, including coronal plane deformity, horizontal plane deformity, and sagittal plane deformity, and concluded that the lateral asymmetry of coronal plane deformity is the best single quantitative indicator of the clinical condition among the three. The experiment described that there is a good correlation between the lateral asymmetry obtained from surface shape analysis and the Cobb angle obtained from X-ray films. Some researchers began to determine the topographic map of the back surface through an experimental device based on structured light projection, and used two topographic variables (axial plane deformity index and posterior trunk symmetry index) to quantify the deformities in two different planes. However, the experimental instrument is greatly affected by ambient light, and the back features are relatively smooth, making it difficult to obtain key point information.

[0006] In summary, a detection system that can quickly obtain the region of interest of scoliosis by using a 3D radiation-free non-contact surface scanning system capable of collecting visual information of the external anatomical structure of the human body, and obtain the lateral symmetry of the back based on the point cloud registration method and the three-dimensional surface reconstruction algorithm, will contribute to the screening of scoliosis. The convenience of system construction and the rapidity of detection methods contribute to the popularization of scoliosis screening, and eliminate the dependence on technicians, avoiding the subjectivity of detection results. Therefore, it is necessary to explore a processing method and system for scoliosis detection based on three-dimensional point cloud features. Summary of the Invention

[0007] The purpose of the present invention is to propose a scoliosis screening method and system based on three-dimensional point cloud information, which can obtain the lateral symmetry of the back based on the point cloud registration method and the three-dimensional surface reconstruction algorithm, quickly obtain the three-dimensional information of the region of interest of scoliosis of the subject and accurately identify its lateral asymmetry, and the model has extremely high accuracy and stability.

[0008] To achieve the above purpose, the technical solution of the present invention is as follows:

[0009] On the one hand, the present invention proposes a scoliosis screening method based on three-dimensional point cloud information, including the following steps:

[0010] S1: Monitor the action execution stage of the subject. After determining that the human body is static based on three-dimensional centroid transformation, obtain the three-dimensional point cloud information of the human back surface, and preprocess the three-dimensional point cloud information of the back surface;

[0011] S2: Based on the preprocessed three-dimensional point cloud information of the back surface, perform a clustering algorithm to obtain the clustering of the human back region. Based on the three-dimensional point cloud of the back region, perform principal component analysis to obtain the initial midline sagittal plane; based on the distance difference of the dorsolateral side in the normal direction of the midline sagittal plane, perform three-dimensional point cloud to two-dimensional image reprojection and polynomial fitting to obtain the region of interest of scoliosis;

[0012] S3: Based on the region of interest of scoliosis, obtain the fitting plane of the human back surface. Initialize the midline sagittal plane as the mirror plane to obtain the mirrored fitting plane and the point cloud of the region of interest of scoliosis. Based on this set of mirrored fitting planes, obtain the Rodriguez rotation matrix. Use the mirrored region of interest of scoliosis as the source point cloud and the region of interest of scoliosis as the target point cloud to complete the rough registration of the source point cloud;

[0013] S4: Based on the target point cloud and the point cloud after rough registration, use the point cloud registration algorithm to obtain the aligned point cloud after fine registration. Based on the aligned point cloud after fine registration and the target point cloud, obtain the mirror symmetry plane of the two point clouds. Map the indices of the corresponding points of the aligned point cloud after fine registration and the target point cloud to the indices in the target point cloud to obtain the symmetric matching points within the target point cloud to get the weak symmetry plane of the back;

[0014] S5: Based on the intersection line and included angle between the mirror symmetry plane and the weak symmetry plane of the back, use the quaternion rotation method to obtain the plane between the two planes to get the set of weak symmetry planes. Based on the set of weak symmetry planes and the symmetry error composite coefficient SECI, use the polynomial interpolation method to obtain the local optimal solution to get the symmetry plane for the analysis of the dorsal transverse asymmetry;

[0015] S6: Based on the symmetry plane for the analysis of the dorsal transverse asymmetry, obtain the projection grid of the region of interest for scoliosis detection. Based on the projection grid, obtain the polynomial surface of the grid point cloud. Based on the polynomial surface, obtain the reprojected grid point cloud and the transverse depth difference point cloud.

[0016] Preferably, the S1 specifically includes:

[0017] Use the first depth camera to continuously and real - time obtain the surface point cloud of the subject during the sub - current flexion test action phase. Use the point cloud registration method to obtain the corresponding points of the subject's back point cloud between the front and rear frames captured by the camera, and calculate the centroid displacement of the corresponding point set. When the centroid displacement is less than the static threshold, enter the measurement phase;

[0018] Use the second depth camera to obtain the three - dimensional point cloud information of the human back surface under static conditions during the measurement phase, and use the pass - through filter to filter out the 3D information (such as impurities like the floor) beyond the threshold range; Use the statistical filtering algorithm to smooth the obtained three - dimensional point cloud information of the back surface to remove the outlier points.

[0019] Preferably, calculating the centroid displacement of the corresponding point set and entering the measurement phase when the centroid displacement is less than the static threshold is specifically:

[0020] Use the first depth camera to obtain the Euclidean distance between the corresponding centroids of the back point cloud of the subject under static conditions multiple times. After removing the discrete points, obtain the upper quartile of the data as the Euclidean distance threshold to obtain the static threshold. When the centroid displacement during the screening process is less than the Euclidean distance threshold, enter the measurement stage.

[0021] Preferably, when the subject stands upright in the screening area, the first depth camera is set at the dorsal side position of the subject, and the second depth camera is set above the subject; the first depth camera is a structured light depth camera, and the second depth camera is a binocular structured light depth camera.

[0022] Preferably, the S2 specifically includes:

[0023] Based on the preprocessed three-dimensional point cloud information of the back surface, use the density-based spatial clustering algorithm DBSCAN to obtain the point cloud clustering information. Based on the prior conditions, the cluster with the largest number is the point cloud set of the back area; use the principal component analysis algorithm PCA to obtain the principal components in three directions of the point cloud set of the back area (three-dimensional point cloud of the back area). Use the principal components as the new basis vectors to obtain the three-dimensional coordinates in the principal component coordinate system. Based on the relative position between the second depth camera and the subject, obtain the initial midline sagittal plane according to the principal component coordinate system;

[0024] Based on the three-dimensional point cloud of the back area in the principal component coordinate system and the internal parameter matrix of the second depth camera (binocular structured light depth camera), use the three-dimensional to two-dimensional projection mapping method to obtain a two-dimensional image. When there is an interference situation where multiple points are mapped to the same pixel position during the mapping process, the pixel RGB uses the color information of the highest depth value;

[0025] Based on the image information mapped from three dimensions to two dimensions, use a morphological closing operation similar to morphology, with a small structure for morphological dilation and a large structure for morphological erosion, to obtain a two-dimensional edge smoothed image;

[0026] Based on the two-dimensional edge smoothed image, calculate the pixel coordinate difference in the dorsolateral pixel coordinate system, use the polynomial fitting method to obtain the seventh-degree polynomial of the dorsolateral pixel coordinate difference, and confirm the upper and lower pixel coordinates at the dorsolateral narrowing according to the gradient relationship after polynomial fitting;

[0027] Based on the upper and lower pixel coordinates at the dorsolateral narrowing, extend along the pixel coordinate axis direction to obtain four pixel coordinates at the junction of the cranial and caudal edges, obtain the minimum value of the cranial pixel coordinates and the maximum value of the caudal pixel coordinates, obtain the bounding box according to the upper and lower pixel thresholds and the cranial and caudal pixel thresholds, map the two pixels at the lower left and upper right corners of the bounding box to the three-dimensional point cloud coordinate system, and obtain four bounding surfaces in the non-depth direction based on the principal component coordinate axis direction to obtain the region of interest for scoliosis.

[0028] Preferably, the S3 specifically includes:

[0029] Based on the region of interest of scoliosis, using the iteratively reweighted least squares method, a fitting plane of the region of interest of scoliosis is obtained, that is, the fitting plane of the human back surface;

[0030] Taking the initialized midline sagittal plane as the mirror plane, obtaining the mirror images of the region of interest of scoliosis and its fitting plane, taking the intersection line of the two mirror-symmetric fitting planes as the rotation axis, and the included angle as the selected rotation angle, using the Rodriguez rotation matrix to obtain the rough registration rotation matrix;

[0031] Taking the region of interest of scoliosis as the target point cloud and the mirror image of the region of interest of scoliosis as the source point cloud, rotating the source point cloud based on the rough registration rotation matrix to obtain the aligned point cloud, which is used as the source point cloud for fine registration.

[0032] Preferably, the S4 specifically includes:

[0033] Based on the target point cloud and the source point cloud for fine registration, using the point cloud rigid registration method, the iteration end condition in the registration process is that the change in the transformation matrix parameters obtained in two adjacent iterations is less than a preset threshold, and after continuously satisfying the threshold condition multiple times (for example, 5 times), it indicates that the transformation has tended to be stable and the iteration is stopped. The change in the transformation matrix parameters includes the change in the rotation angle and the change in the translation amount; after the iteration ends, the aligned point cloud for fine registration is obtained;

[0034] Based on the aligned point cloud for fine registration and the target point cloud, the corresponding points between the aligned point cloud for fine registration and the target point cloud are obtained according to the minimum Euclidean distance, and the mirror-symmetric plane is obtained from the centroids of the two point clouds. Since the index of the points in the point cloud is not changed during the mirroring process, the index of the corresponding points is directly mapped to the index of the weak symmetric point pairs inside the target point cloud. Based on the combination of the included angle between the weak symmetric point pairs and the fitting plane and the distance between the points in the weak symmetric point pairs and the fitting plane as the iteration condition, the weak symmetric plane of the back is obtained based on the RANSAC algorithm.

[0035] Preferably, the S5 specifically includes:

[0036] Based on the intersection line and the included angle between the mirror-symmetric plane and the weak symmetric plane of the back, taking the intersection line as the rotation axis and the included angle as the rotation angle, dividing the rotation angle range into multiple rotation angles equally to obtain a rotation angle set;

[0037] Based on the rotation axis and the rotation angle set, using the quaternion rotation method, a quaternion rotation set is obtained, and the mirror-symmetric plane is rotated to obtain a weak symmetric plane set including the mirror-symmetric plane and the weak symmetric plane of the back;

[0038] Based on the weakly symmetric plane set, the symmetric error composite coefficient SECI of the weakly symmetric planes in the set is calculated. SECI is a combination of the average Euclidean distance R of corresponding points and the projection area coverage rate P. After mirroring the region of interest of scoliosis with respect to the weakly symmetric plane, a mirrored point cloud is obtained. The nearest neighbor points between the mirrored point cloud and the point cloud of the region of interest of scoliosis are a set of corresponding points. The average Euclidean distance of the corresponding points is calculated. The mirrored point cloud and the point cloud of the region of interest of scoliosis are projected onto the plane fitted by the iteratively reweighted least squares method for the region of interest of scoliosis. The convex hull algorithm is used to obtain the minimum convex polygon of the projected point cloud. The projection area coverage rate is obtained based on the overlapping area and the projected area of the point cloud of the region of interest of scoliosis. The local optimal symmetric plane is obtained based on the interpolation optimal solution of SECI;

[0039]

[0040] In the formula, P i represents a point in the point cloud of the region of interest of scoliosis, Q i represents the corresponding point in the mirrored point cloud corresponding to P i The number of corresponding point pairs is N, C(T) and C(A) respectively represent the convex hull areas corresponding to the point cloud of the region of interest of scoliosis and the mirrored point cloud, and S represents the symmetric error composite coefficient SECI.

[0041] Preferably, the specific steps of S6 include:

[0042] Using the unit normal vector of the local optimal symmetric plane as the unit axis vector of the new coordinate system, applying the Smith orthogonalization method to process the depth direction axis vector of the principal component coordinate system, taking the centroid of the point cloud of the region of interest of scoliosis as the origin, and obtaining the re-interpolation coordinate system with the vector product of the two unit axis vectors as the third axis vector for the re-interpolation after the surface reconstruction of the point cloud. After reconstructing the coordinate system, coordinate transformation is performed; Based on the region of interest of scoliosis and its mirrored point cloud with respect to the local optimal symmetric plane, the three-dimensional information is mapped to the plane of the re-interpolation coordinate system perpendicular to the local optimal symmetric plane to obtain the axis-aligned bounding box of the two-dimensional point cloud in the plane. The resolution of the point cloud within the region of interest of scoliosis is calculated, and a two-dimensional grid is generated based on the resolution;

[0043] Based on the three-dimensional point cloud of the back region in the principal component coordinate system in S2 and the local optimal symmetric plane, the mirrored point cloud of the three-dimensional point cloud of the back region is obtained. Based on the three-dimensional point cloud of the back region and the two-dimensional grid, the three-dimensional point cloud of the back region is projected onto the plane where the two-dimensional grid is located. Taking the grid points as the center points, a distance threshold is set to obtain the neighboring points of the projected points corresponding to the grid points. The polynomial surface fitting method is used to obtain the surface fitted by the neighboring points, and the depth value of the grid points is calculated. Repeat the above steps in sequence to obtain the point cloud of the region of interest of scoliosis and its mirrored re-projected point cloud, and obtain the lateral depth difference point cloud according to the re-projected point cloud.

[0044] On the one hand, the present invention provides a scoliosis screening system based on three-dimensional point cloud information. The system is implemented by using the scoliosis screening method based on three-dimensional point cloud information as described above, and includes a point cloud data acquisition module, a scoliosis region acquisition module, a point cloud registration module, and a lateral asymmetry analysis module;

[0045] The point cloud data acquisition module uses a depth camera to monitor the dynamics of the subject and obtain the three-dimensional point cloud information of the back;

[0046] The scoliosis region acquisition module uses a density-based spatial clustering algorithm and a point cloud reprojection algorithm to obtain a clustered two-dimensional map, and obtains the region of interest through an image morphology processing algorithm and a polynomial curve fitting method;

[0047] The point cloud registration module uses the iteratively reweighted least squares method and the principal component analysis algorithm to obtain symmetric point clouds, and performs a point cloud registration algorithm after rough registration using the Rodriguez rotation matrix;

[0048] The lateral asymmetry analysis module uses the registered point clouds to obtain a symmetry plane, and uses a surface reconstruction algorithm to obtain the lateral depth difference point clouds.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] The present invention provides a simple and efficient static monitoring method, which effectively reduces the manual participation in the screening process and avoids the influence of subjective factors;

[0051] The present invention improves the method for obtaining the region of interest in scoliosis, uses the principal component analysis technology to improve the influence of the shooting angle on the field of view of the detection system, and uses the morphological closing operation processing to reduce the influence of noise and occlusion;

[0052] The present invention uses a preprocessing method based on the position relationship of the fitting plane to avoid the situation where the point cloud falls into a local optimum;

[0053] The present invention proposes the SECI coefficient, strengthens the characteristics of internal symmetry, reduces the dependence on the rigid registration method, and enhances the dependence on the symmetry plane acquisition method based on registration;

[0054] The present invention improves the interpolation method for obtaining the lateral depth difference point clouds, uses the point clouds within the interpolation point distance threshold for polynomial fitting to obtain the surface equation, and reduces the influence of too many points and too few points during interpolation. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is the flow chart of the method of the present invention;

[0056] Figure 2 is the schematic diagram of the system principle of the present invention Figure 1 ;

[0057] Figure 3 Schematic diagram of the system principle of the present invention Figure 2 ;

[0058] Figure 4 Schematic diagram of the system principle of the present invention Figure 3 。 Specific implementation manners

[0059] Next, in combination with the attached Figures 1-4 , the technical solution of the present invention will be specifically described.

[0060] Please refer to Figure 1 , the present invention is a scoliosis screening method and system based on three-dimensional point cloud information, and the method includes the following steps:

[0061] Step S1: Monitor the action execution stage of the subject. After judging that the human body is in a static state based on three-dimensional centroid transformation, obtain the three-dimensional point cloud information of the human back surface, and preprocess the three-dimensional point cloud information of the back surface;

[0062] Step S2: Based on the preprocessed three-dimensional point cloud information of the back surface, perform a clustering algorithm to obtain the clustering of the human back region. Based on the three-dimensional point cloud of the back region, perform principal component analysis to obtain the initial midline sagittal plane; Based on the distance difference of the dorsolateral side in the normal direction of the midline sagittal plane, perform three-dimensional point cloud to two-dimensional image reprojection and polynomial fitting to obtain the region of interest for scoliosis;

[0063] Step S3: Based on the region of interest for scoliosis, obtain the fitting plane of the human back surface. Using the initial midline sagittal plane as the mirror plane, obtain the mirror-symmetrical fitting plane and the point cloud of the region of interest for scoliosis. Based on this set of mirror-symmetrical fitting planes, obtain the Rodriguez rotation matrix. Using the mirror image of the region of interest for scoliosis as the source point cloud and the region of interest for scoliosis as the target point cloud, complete the rough registration of the source point cloud;

[0064] Step S4: Based on the target point cloud and the point cloud after rough registration, use the point cloud registration algorithm to obtain the finely registered aligned point cloud. Based on the finely registered aligned point cloud and the target point cloud, obtain the mirror-symmetrical plane of the two point clouds. Map the indexes of the corresponding points of the finely registered aligned point cloud and the target point cloud to the indexes in the target point cloud, and obtain the symmetric matching points in the target point cloud to obtain the weak symmetric plane of the back;

[0065] Step S5: Based on the intersection line and included angle of the mirror-symmetrical plane and the weak symmetric plane of the back, use the quaternion rotation method to obtain the plane between the two planes to obtain the set of weak symmetric planes. Based on the set of weak symmetric planes and the symmetric error composite coefficient SECI, use the polynomial interpolation method to obtain the local optimal solution to obtain the symmetric plane for the analysis of the dorsal transverse asymmetry;

[0066] Step S6: Based on the symmetry plane analyzed from the lateral back asymmetry degree, obtain the projection grid of the region of interest for scoliosis detection. Based on the projection grid, obtain the polynomial surface of the grid point cloud. Based on the polynomial surface, obtain the reprojected grid point cloud and the lateral depth difference point cloud.

[0067] The following is the specific implementation process of the present invention.

[0068] In this embodiment, step S1 is specifically as follows:

[0069] The first depth camera uses a pass-through filter to remove 3D information outside the threshold range of 0.3 - 2.0 meters, and continuously obtains the surface point cloud of the moving subject in real time. The SOR filter is used on the surface point cloud to remove noise points and outliers and maintain the integrity of the obtained surface point cloud. The first stable frame captured by the first depth camera is used as the source point cloud for alignment, and the subsequent stable frames are used as the target point cloud. The point-to-point iterative closest point algorithm is used to obtain the corresponding points between two consecutive frames, and the Euclidean distance between the centroids of the corresponding point groups is obtained.

[0070] According to the established system, the Euclidean distance between the corresponding points of the subject in the static state is obtained multiple times using the first depth camera. After removing the discrete points, the upper quartile of the data is obtained as the static threshold for this screening. When the threshold condition is met during the screening process, the second depth camera is used to obtain the three-dimensional point cloud information of the back surface in the static state, and the pass-through filter is used to remove the 3D information outside the threshold range of 0.3 - 1.7 meters.

[0071] In an embodiment of the present invention, the static monitoring of the subject is specifically as follows:

[0072] After the system is built and the coverage range of the visual sensor is determined, the subject advances from outside the field of view range to within the coverage range of the system in sequence. After reaching the specified position, the subject performs the forward flexion action in the sub-current forward flexion test, and remains in the forward flexion state unchanged after completion. The system static monitoring is to obtain the three-dimensional information of the back surface in the forward flexion state without manual interference.

[0073] In this embodiment, step S2 is specifically as follows:

[0074] According to the obtained three-dimensional point cloud information of the back surface in the forward flexion state, the three-dimensional point cloud data of the human back region is obtained using the density-based spatial clustering algorithm and the statistical outlier removal filter. The three principal components of the point cloud are obtained using the principal component analysis algorithm, and a new coordinate system is defined with the principal components for coordinate transformation of the three-dimensional point cloud data of the human back region. According to the pinhole imaging principle of the camera, the mapping relationship from the three-dimensional point cloud to the two-dimensional image is obtained using the internal and external parameters of the second depth camera to obtain the reprojected image information.

[0075] Using a method similar to morphological closing operation based on the image information after reprojection, and obtaining a hole-free image with different structural sizes of morphological dilation and erosion. After obtaining a grayscale image using the adaptive threshold method, a binary image is obtained using Otsu's method. The Sobel operator is used to detect the gradient in the Y direction in the pixel coordinate system to obtain the outer boundary of the back. According to the pixel difference of the outer boundary, a seventh-order curve is obtained using polynomial fitting based on the least squares method. According to the gradient relationship after polynomial fitting, the upper and lower pixel coordinates at the narrowing of the dorsolateral part are found.

[0076] Based on the upper and lower pixel coordinates at the narrowing of the dorsolateral part, extend along the pixel coordinate axis direction to obtain four pixel coordinates at the junction of the cranial and caudal edges. Obtain the minimum value of the cranial pixel coordinates and the maximum value of the caudal pixel coordinates. According to the upper and lower pixel thresholds and the cranial-caudal pixel thresholds, obtain the bounding box. Map the two pixels at the lower left corner and the upper right corner of the bounding box to the three-dimensional point cloud coordinate system. Based on the axis direction of the principal component coordinate system passing through the two points, obtain four enclosing surfaces in the non-depth direction to obtain the region of interest for scoliosis.

[0077] In an embodiment of the present invention, the reprojection method is specifically as follows:

[0078] Based on the three-dimensional point cloud information of the back in the principal component coordinate system and the internal and external parameter matrices of the second depth camera, a two-dimensional image is obtained using a three-dimensional to two-dimensional projection mapping method. When there is an interference situation where multiple points are mapped to the same pixel position during the mapping process, the pixel RGB uses the color information of the highest depth value to solve the occlusion problem after coordinate transformation.

[0079] In this embodiment, step S3 is specifically as follows:

[0080] Using the iteratively reweighted least squares method, obtain the fitting plane of the region of interest for scoliosis. Use the axis direction in the principal component coordinate system and the centroid coordinates of the point cloud in the region of interest for scoliosis to obtain the initialized midline sagittal plane. Use the initialized midline sagittal plane as the mirror plane to obtain the mirror image of the region of interest for scoliosis and its fitting plane. According to the two mirror-symmetric planes, use the intersection line of the two planes as the rotation axis and the included angle as the selected rotation angle to obtain the Rodriguez rotation matrix, which is the rough registration rotation matrix. Use the region of interest for scoliosis as the target point cloud and its mirror image as the source point cloud, and use the rough registration rotation matrix to obtain the roughly registered and aligned point cloud.

[0081] In this embodiment, step S4 is specifically as follows:

[0082] Taking the aligned point cloud after coarse registration as the source point cloud for fine registration, using the rigid point cloud registration method, the iteration end condition during the registration process is that the changes in the transformation matrix parameters (changes in rotation angle and translation amount) obtained in two adjacent iterations are less than the preset threshold, and after continuously satisfying the threshold condition 5 times, it indicates that the transformation has tended to be stable and the iteration stops. After the iteration ends, the aligned point cloud after fine registration is obtained, and the corresponding point set between the aligned point cloud and the target point cloud is obtained using the kd-tree search method according to the minimum Euclidean distance.

[0083] Using the centroids of the aligned point cloud after fine registration and the target point cloud to obtain the symmetric plane of the two centroids, and taking it as the mirror symmetric plane of the target point cloud. Since the index of the points in the point cloud is not changed during the mirroring process in step S3, the index of the corresponding points is directly mapped to the inside of the target point cloud to obtain the central point set of the corresponding points inside the target point cloud, and the RANSAC algorithm is used to obtain the fitting plane of the central point set, and taking it as the weak symmetric plane of the target point cloud.

[0084] In this embodiment, step S5 is specifically as follows:

[0085] Taking the intersection line of the mirror symmetric plane and the weak symmetric plane as the rotation axis, and the included angle as the rotation angle. Within the range of the rotation angle, it is evenly divided to obtain multiple subdivision angles with equal angular spans, and taking them as the rotation angle set. Let the angle threshold range be [θ min , θ max , and the number of equal parts is n, then the angular width of each interval is Using the quaternion rotation method, a quaternion rotation set is obtained according to the rotation angle set, so that the mirror symmetric plane rotates towards the weak symmetric plane to obtain a symmetric plane set including the mirror symmetric plane and the weak symmetric plane.

[0086] Calculate the symmetric error composite coefficient (SECI) of each weak symmetric plane in the weak symmetric plane set. SECI is the combination of the average Euclidean distance of the corresponding points and the projection area coverage rate. After mirroring the region of interest in scoliosis with respect to the weak symmetric plane, the mirror point cloud is obtained, and the nearest neighbor point in the mirror point cloud and the point cloud of the region of interest in scoliosis is a group of corresponding points to calculate the average Euclidean distance of the corresponding points. Project the mirror point cloud and the point cloud of the region of interest in scoliosis onto the plane fitted by the iteratively reweighted least squares method in the region of interest in scoliosis, use the convex hull algorithm to obtain the minimum convex polygon of the projected point cloud, and obtain the projection area coverage rate according to the overlapping area and the projected area of the point cloud of the region of interest in scoliosis. Based on the interpolation optimal solution of SECI, the local optimal symmetric plane is obtained.

[0087] In this embodiment, step S6 is specifically as follows:

[0088] Use the unit normal vector of the local optimal symmetry plane as the unit axis vector of the new coordinate system, apply the Smith orthogonalization method to process the depth direction axis vector of the principal component coordinate system, take the centroid of the point cloud in the region of interest of scoliosis as the origin, and obtain the re-interpolation coordinate system with the vector product of the two unit axis vectors as the third axis vector for re-interpolation after point cloud surface reconstruction. After reconstructing the coordinate system, perform coordinate transformation. Map the three-dimensional information of the region of interest of scoliosis and its mirror point cloud with respect to the local optimal symmetry plane to the plane of the re-interpolation coordinate system perpendicular to the local optimal symmetry plane to obtain the axis-aligned bounding box of the two-dimensional point cloud in the plane, calculate the resolution of the point cloud in the region of interest of scoliosis, and perform equidistant division based on the resolution and the length and width of the axis-aligned bounding box. If the number of grids divided according to the resolution cannot be rounded up, round up, and generate a two-dimensional grid for the axis-aligned bounding box.

[0089]

[0090] The above formula is the Smith orthogonalization process, where α1 and α2 represent two non-orthogonal vectors, and β1, β2, and β3 represent the vectors after orthogonalization. β1, β2, and β3 can represent the direction vectors of the coordinate system; after calculating the direction vectors, perform unitization.

[0091] Map the three-dimensional point cloud data of the human back region obtained by the density-based spatial clustering algorithm and the statistical outlier removal filter in step S2 to the re-interpolation coordinate system, project the three-dimensional point cloud of the back in the re-interpolation coordinate system onto the plane where the two-dimensional grid is located, take the grid point as the center point, set the distance threshold to obtain the neighboring points of the projection point corresponding to the grid point, use the polynomial surface fitting method to obtain the surface fitted by the neighboring points, calculate the depth value of the grid point according to the polynomial equation, and calculate the point cloud after re-interpolation in the region of interest of scoliosis in turn. Obtain the mirror image of the three-dimensional point cloud of the back in the re-interpolation coordinate system according to the local optimal symmetry plane, and repeat the above process to obtain the point cloud after re-interpolation of the mirror image in the region of interest of scoliosis. Calculate the depth difference between the two sets of point clouds at the same grid point to obtain the lateral depth difference point cloud.

[0092]

[0093] Among them, I asym represents the mean value of the depth difference point cloud, d i,j represents the depth difference corresponding to the two-dimensional grid, i and j represent the horizontal and vertical coordinates of the two-dimensional grid, and |G v | represents the number of valid grid points. If the back shape of |G v | is completely symmetric with respect to the local optimal symmetry plane, the I asym exponent is zero, and the more the back shape deviates from perfect symmetry, the larger Iasym is.

[0094] The above are the preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention in terms of the functions and effects produced shall fall within the protection scope of the present invention.

Claims

1. A scoliosis screening method based on three-dimensional point cloud information, characterized in that, The following steps are involved: S1: During the monitoring of the subject's action execution phase, after judging that the human body is in a static state based on the three-dimensional centroid transformation, the three-dimensional point cloud information of the human back surface is obtained, and the three-dimensional point cloud information of the back surface is preprocessed; S2: Based on the preprocessed three-dimensional point cloud information of the back surface, a clustering algorithm is used to obtain the clustering of the human back area. Based on the three-dimensional point cloud of the back area, principal component analysis is performed to obtain the initialization midline sagittal plane; Based on the distance difference of the dorsolateral part in the normal direction of the midline sagittal plane, the 3D point cloud is reprojected into a 2D image and polynomial fitting is performed to obtain the scoliosis region of interest; S3: Based on the scoliosis region of interest, obtain the fitting plane of the back surface of the human body, take the initial midline sagittal plane as the mirror plane, obtain the mirror-symmetric fitting plane and the point cloud of the scoliosis region of interest, and based on this set of mirror-symmetric fitting planes, obtain the Rodriguez rotation matrix, take the mirror image of the scoliosis region of interest as the source point cloud, take the scoliosis region of interest as the target point cloud, and complete the coarse registration of the source point cloud; S4: Based on the target point cloud and the point cloud after coarse registration, a point cloud registration algorithm is used to obtain an aligned point cloud after fine registration. Based on the aligned point cloud after fine registration and the target point cloud, a mirror symmetry surface of the two point clouds is obtained. The indexes of the corresponding points of the aligned point cloud after fine registration and the target point cloud are mapped to the indexes in the target point cloud, and the symmetric matching points in the target point cloud are obtained to obtain the weak symmetry surface of the back. S5: Based on the intersection line and angle between the mirror symmetry plane and the back weak symmetry plane, the quaternion rotation method is used to obtain the plane between the two planes to obtain the weak symmetry plane set. Based on the weak symmetry plane set and the symmetry error composite coefficient SECI, the polynomial interpolation method is used to obtain the local optimal solution to obtain the symmetry plane for the back lateral asymmetry analysis; S6: Based on the symmetry plane of the back transverse asymmetry analysis, the projection grid of the region of interest for scoliosis detection is obtained. Based on the projection grid, the polynomial surface of the grid point cloud is obtained. Based on the polynomial surface, the reprojected grid point cloud and the transverse depth difference point cloud are obtained.

2. The scoliosis screening method based on three-dimensional point cloud information according to claim 1, wherein The S1 specifically includes: The first depth camera is used to continuously acquire the surface point cloud of the subject in real time when the subject is performing the forward flexion test action stage, and the corresponding points of the point cloud of the subject's back between the front and rear frames acquired by the camera are obtained using the point cloud registration method, and the centroid displacement of the corresponding point set is calculated. When the centroid displacement is less than the static threshold, the measurement stage is entered; The second depth camera is used to obtain the three-dimensional point cloud information of the human back surface in static state during the measurement phase, and a pass-through filter is used to filter out the 3D information beyond the threshold range. The statistical filtering algorithm is used to smooth the obtained three-dimensional point cloud information of the back surface and remove outliers.

3. The scoliosis screening method based on three-dimensional point cloud information according to claim 2, wherein Calculate the centroid displacement of the corresponding point set. When the centroid displacement is less than the static threshold, enter the measurement phase, which is as follows: The first depth camera is used to obtain the Euclidean distance between the center of mass of corresponding points on the static point cloud of the subject's back multiple times. After removing discrete points, the upper quartile of the data is obtained as the Euclidean distance threshold to obtain the static threshold. When the displacement of the center of mass during the screening process is less than the Euclidean distance threshold, the measurement phase is entered.

4. A scoliosis screening method based on three-dimensional point cloud information according to claim 2, characterized in that, When the subject stands upright within the screening area, the first depth camera is set at the dorsal side position of the subject, and the second depth camera is set at the upper position of the subject; the first depth camera is a structured light depth camera, and the second depth camera is a binocular structured light depth camera.

5. A scoliosis screening method based on three-dimensional point cloud information according to claim 1, characterized in that, Specifically, S2 includes: Based on the preprocessed three-dimensional point cloud information of the back surface, the density-based spatial clustering algorithm DBSCAN is used to obtain the point cloud clustering information. Based on the prior conditions, the cluster with the largest number is the point cloud set of the back region; the principal component analysis algorithm PCA is used to obtain the principal components in three directions of the point cloud set of the back region, and the three-dimensional coordinates in the principal component coordinate system are obtained with the principal components as the new basis vectors. Based on the relative position between the second depth camera and the subject, the initial midline sagittal plane is obtained according to the principal component coordinate system; Based on the three-dimensional point cloud of the back region in the principal component coordinate system and the internal parameter matrix of the second depth camera, the three-dimensional to two-dimensional projection mapping method is used to obtain a two-dimensional image. When there is an interference situation where multiple points are mapped to the same pixel position during the mapping process, the pixel RGB uses the color information of the highest depth value; Based on the image information mapped from three dimensions to two dimensions, the morphological closing operation is used. The structure of morphological dilation is small, and the structure of morphological erosion is large to obtain a two-dimensional edge smoothed image; Based on the two-dimensional edge smoothed image, the pixel coordinate differences in the dorsolateral pixel coordinate system are calculated, and the polynomial fitting method is used to obtain the seventh-degree polynomial of the dorsolateral pixel coordinate differences. According to the gradient relationship after polynomial fitting, the upper and lower pixel coordinates at the dorsolateral narrowing are confirmed; Based on the upper and lower pixel coordinates at the dorsolateral narrowing, extend along the pixel coordinate system axis direction to obtain four pixel coordinates at the junction of the cranial and caudal edges, obtain the minimum value of the cranial pixel coordinates and the maximum value of the caudal pixel coordinates, obtain the bounding box according to the upper and lower pixel thresholds and the cranial and caudal pixel thresholds, map the two pixels at the lower left corner and the upper right corner of the bounding box to the three-dimensional point cloud coordinate system, and obtain four bounding surfaces in the non-depth direction based on the principal component coordinate system axis direction to obtain the region of interest for scoliosis.

6. The scoliosis screening method based on three-dimensional point cloud information according to claim 1, characterized in that Specifically, S3 includes: Based on the region of interest for scoliosis, the iteratively reweighted least squares method is used to obtain the fitting plane of the region of interest for scoliosis, that is, the fitting plane of the human back surface; Taking the initial midline sagittal plane as the mirror plane, obtain the mirror image of the region of interest for scoliosis and its fitting plane. Taking the intersection line of the two mirror-symmetric fitting planes as the rotation axis and the included angle as the selected rotation angle, use the Rodriguez rotation matrix to obtain the rough registration rotation matrix; Taking the region of interest for scoliosis as the target point cloud and the mirror image of the region of interest for scoliosis as the source point cloud, rotate the source point cloud based on the rough registration rotation matrix to obtain the aligned point cloud, which is used as the source point cloud for fine registration.

7. A scoliosis screening method based on three-dimensional point cloud information according to claim 6, characterized in that Specifically, S4 includes: Based on the target point cloud and the source point cloud after fine registration, using the point cloud rigid registration method, the end condition of the iteration during the registration process is that the change in the transformation matrix parameters obtained in two adjacent iterations is less than a preset threshold, and after continuously satisfying the threshold condition multiple times, it indicates that the transformation has tended to be stable, and the iteration is stopped. The change in the transformation matrix parameters includes the change in the rotation angle and the change in the translation amount; after the iteration ends, the aligned point cloud after fine registration is obtained. Based on the aligned point cloud after fine registration and the target point cloud, the corresponding points between the aligned point cloud after fine registration and the target point cloud are obtained according to the minimum Euclidean distance. The mirror symmetry plane is obtained from the centroids of the two point clouds. Since the index of the points in the point cloud is not changed during the mirroring process, the index of the corresponding points is directly mapped to the index of the weak symmetric point pairs inside the target point cloud. Based on the combination of the angle between the weak symmetric point pairs and the fitting plane and the distance from the points in the weak symmetric point pairs to the fitting plane as the iteration condition, the random sample consensus algorithm RANSAC is used to obtain the weak symmetric plane of the back.

8. The scoliosis screening method based on three-dimensional point cloud information according to claim 1, wherein The specific steps of S5 are as follows: Based on the intersection line and the included angle between the mirror symmetry plane and the weak symmetric plane of the back, with the intersection line as the rotation axis and the included angle as the rotation angle, the rotation angle range is equally divided into multiple rotation angles to obtain a set of rotation angles; Based on the rotation axis and the set of rotation angles, using the quaternion rotation method, a set of quaternion rotations is obtained, and the mirror symmetry plane is rotated to obtain a set of weak symmetric planes including the mirror symmetry plane and the weak symmetric plane of the back; Based on the set of weak symmetric planes, calculate the symmetry error composite coefficient SECI of the weak symmetric planes in the set. SECI is the synthesis of the average Euclidean distance R of the corresponding points and the projection area coverage rate P. The region of interest of scoliosis is mirrored about the weak symmetric plane to obtain the mirrored point cloud. The nearest neighbor points in the mirrored point cloud and the point cloud of the region of interest of scoliosis are a set of corresponding points. Calculate the average Euclidean distance of the corresponding points. Project the mirrored point cloud and the point cloud of the region of interest of scoliosis onto the plane fitted by the iteratively reweighted least squares method in the region of interest of scoliosis. Use the convex hull algorithm to obtain the minimum convex polygon of the projected point cloud. Calculate the projection area coverage rate according to the overlapping area and the projected area of the point cloud of the region of interest of scoliosis. Based on the interpolation optimal solution of SECI, obtain the local optimal symmetric plane; S = P R -R Wherein, P i represents a point in the point cloud of the region of interest of scoliosis, Q i represents the corresponding point in the mirror point cloud corresponding to P i The number of corresponding point pairs is represented by N, C(T) and C(A) respectively represent the convex hull areas corresponding to the point cloud of the region of interest of scoliosis and the mirror point cloud, and S represents the symmetric error composite coefficient SECI.

9. The scoliosis screening method based on three-dimensional point cloud information according to claim 8, characterized in that, The specific steps of S6 are as follows: Use the unit normal vector of the local optimal symmetric plane as the unit axis vector of the new coordinate system, and apply the Smith orthogonalization method to process the depth direction axis vector of the principal component coordinate system. With the centroid of the point cloud of the region of interest of scoliosis as the origin, obtain the third unit axis vector by the vector product of the two unit axis vectors to obtain the re-interpolation coordinate system for the re-interpolation after the surface reconstruction of the point cloud. After reconstructing the coordinate system, perform coordinate transformation; Based on the region of interest of scoliosis and its mirrored point cloud about the local optimal symmetric plane, map the three-dimensional information to the plane of the re-interpolation coordinate system perpendicular to the local optimal symmetric plane to obtain the axis-aligned bounding box of the two-dimensional point cloud in the plane. Calculate the resolution of the point cloud within the region of interest of scoliosis, and generate a two-dimensional grid based on the resolution; Based on the three-dimensional point cloud of the back region and the locally optimal symmetry plane in the principal component coordinate system in S2, the mirror point cloud of the three-dimensional point cloud of the back region is obtained. Based on the three-dimensional point cloud of the back region and the two-dimensional grid, the three-dimensional point cloud of the back region is projected onto the plane where the two-dimensional grid is located. Taking the grid points as the center points, a distance threshold is set to obtain the neighboring points of the projection points corresponding to the grid points. The polynomial surface fitting method is used to obtain the surface fitted by the neighboring points, and the depth value of the grid points is calculated. This process is repeated successively to obtain the point cloud of the region of interest in scoliosis and its mirror re-projected point cloud, and the lateral depth difference point cloud is obtained based on the re-projected point cloud. A scoliosis screening system based on three-dimensional point cloud information, characterized in that, The system is implemented using the scoliosis screening method based on three-dimensional point cloud information described in any one of claims 1-9, and includes a point cloud data acquisition module, a scoliosis region acquisition module, a point cloud registration module, and a lateral asymmetry analysis module; The point cloud data acquisition module uses a depth camera to monitor the dynamics of the subject and obtain the three-dimensional point cloud information of the back; The scoliosis region acquisition module uses the density-based spatial clustering algorithm and the point cloud re-projection algorithm to obtain the clustered two-dimensional map, and obtains the region of interest through the image morphology processing algorithm and the polynomial curve fitting method; The point cloud registration module uses the iteratively reweighted least squares method and the principal component analysis algorithm to obtain the symmetric point cloud, and performs the point cloud registration algorithm after rough registration using the Rodriguez rotation matrix; The lateral asymmetry analysis module uses the registered point cloud to obtain the symmetry plane, and uses the surface reconstruction algorithm to obtain the lateral depth difference point cloud.

Images