Teenager scoliosis detection method based on three-dimensional imaging

Abstract

The invention belongs to the technical field of image processing, and mainly relates to a teenager scoliosis detection method based on three-dimensional imaging, and the method comprises the steps: obtaining multi-modal data such as a back depth map, temperature distribution, an attitude angle and a plantar pressure matrix of a detected person through a depth camera, an infrared imaging device, an I MU sensor and a plantar pressure distribution sensor; the cloud platform generates a three-dimensional point cloud and obtains a back three-dimensional curved surface model by adopting a Poisson surface reconstruction algorithm, overall attitude correction is carried out by combining an attitude angle and pressure gravity center shift, and a thermal correction model of local deformation is constructed based on a thermodynamic diagram; automatically identifying spine segment mark points through a deep learning algorithm, fitting a spine center line, calculating a Cobb angle, judging the scoliosis degree and outputting an analysis report; according to the method, X-ray imaging is not needed, the detection safety is high, attitude, mechanical and thermal information can be fused, artificial attitude interference is reduced, and the method is suitable for early screening and dynamic reexamination of teenagers.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and mainly relates to a juvenile scoliosis detection method based on three-dimensional imaging. BACKGROUND

[0002] Juvenile scoliosis is a common spinal deformity disease during growth and development, which is characterized by abnormal lateral curvature of the spine in the coronal plane, often accompanied by spinal rotation and torso asymmetry, and may lead to bone deformation, limited heart and lung function, and a series of psychological problems.

[0003] In the image processing method and system for juvenile scoliosis detection with Chinese application number CN202410197322.3, the application relates to the technical field of image processing, and discloses an image processing method and system for juvenile scoliosis detection, which comprises obtaining a training set of a classification model, training the classification model, obtaining a trained classification model, inputting a spine curvature image to be classified into the trained classification model, and classifying the spine curvature image; inputting a spine curvature image to be predicted into a trained deep learning model to predict the spine center point; the spine curvature image includes a single spine curvature image, a double spine curvature image and a triple spine curvature image; different models are selected according to different image structures to predict the spine center point, so that the output spine center point is more accurate, the difficulty of predicting the center point of the curved segment of the spine is overcome, and the calculation accuracy of the Cobb angle is improved.

[0004] The above method and the image processing technology can perfectly judge the scoliosis condition, analyze the image of the juvenile spine through image processing technology, and calculate the Cobb angle to evaluate the degree of scoliosis; however, in the existing image processing technology, the X-ray image of the back of the juvenile is usually used as input data, although the single radiation dose is small and the detection frequency is limited, there is still a certain radiation risk, especially for the juvenile group in the growth and development stage, the potential influence of radiation cannot be ignored; in addition, the existing detection method often only relies on a single image for analysis, but in actual situations, juveniles may actively hide the real spinal abnormalities by adjusting the body center of gravity and controlling the body posture in an unobvious way due to psychological reasons, resulting in unobvious manifestation of the degree of scoliosis in the image, which ultimately affects the accuracy of diagnosis and even misses the best opportunity for intervention; therefore, there is an urgent need for a scoliosis detection method based on three-dimensional imaging without the support of X-ray images, which can fuse multiple data sources to improve the detection accuracy, so as to more safely and efficiently serve the early screening and dynamic evaluation of the juvenile group. SUMMARY

[0005] The application provides a kind of detection method of juvenile scoliosis based on three-dimensional imaging, to solve the problem of radiation risk caused by the dependence of X-ray imaging in the existing detection means, and the problem that image analysis is easily disturbed by the active adjustment of the posture of the detected person.

[0006] In order to solve the above problems, the application uses the following technology to realize:

[0007] A kind of detection method of juvenile scoliosis based on three-dimensional imaging:

[0008] A variety of sensors are set to obtain the body data of the detected person, to guide the detected person to perform data collection work in a standard posture, to obtain original data, and to upload the original data obtained by the sensors to a cloud platform;

[0009] The cloud platform pre-processes the original data obtained by the sensors to convert them into standardized data; analyzes the standardized data to obtain a three-dimensional point cloud data set for modeling, posture correction data and pressure correction data;

[0010] According to the three-dimensional point cloud data set, further effective point screening is performed, and a back three-dimensional surface model is reconstructed using a Poisson surface reconstruction algorithm; the back three-dimensional surface model is corrected in posture in combination with the posture correction data; a human body thermal map is introduced to construct a back thermal correction model; according to the back three-dimensional surface model, a landmark recognition algorithm is used to extract a spinal segment landmark to obtain a spinal midline form; based on the generated spinal midline and back thermal correction model, the spinal midline is corrected in thermal; the spinal midline is divided into upper and lower vertebral bodies, the Cobb angle is calculated, the scoliosis condition is judged, and a report is output.

[0011] As a preferred embodiment, the original data acquisition specifically includes:

[0012] A variety of sensors are set at the detection point, the sensor types include depth camera, infrared imaging device, IMU sensor and plantar pressure distribution sensor, the depth camera is used to obtain the back depth map of the detected person, the infrared imaging device is used to obtain the back temperature data of the detected person, the IMU sensor is used to obtain the body posture data of the detected person, and the plantar pressure distribution sensor is used to obtain the plantar pressure data of the detected person;

[0013] The detected person is guided to perform data collection work in a standard posture, and the specific requirements are: the room temperature is lower than 28℃, the detected person is guided to stand on the plantar pressure distribution sensor in the detection area, the feet are parallel and apart, the back is against the depth camera and the infrared imaging device, and the IMU sensor is worn on the detected person's back midline near the seventh cervical vertebra to the first thoracic vertebra region;

[0014] The obtained original data is uploaded to the cloud platform through wireless network.

[0015] As a preferred implementation, the converting into standardized data specifically comprises:

[0016] The complete image data collected by the depth camera and the thermal image collected by the infrared thermal imaging device are subjected to optical correction algorithm based on intrinsic matrix for image distortion correction, and the image is subjected to proportional unification and edge cropping, background interference is removed, and Gaussian filter image noise reduction algorithm is applied to reduce photosensitive device noise interference.

[0017] The original angular velocity and acceleration data in the body posture data collected by the IMU sensor are subjected to Kalman filter algorithm to reduce high-frequency jitter and zero-point drift, and the starting redundant data in the collection process is removed, and the output dimension and format are unified.

[0018] The low-value noise points in the plantar pressure matrix in the plantar pressure distribution data are subjected to threshold removal to reduce non-contact area interference, and the left and right foot pressure maps are unified to the standard template coordinate system.

[0019] As a preferred implementation, the analyzing the standardized data specifically comprises:

[0020] The depth map image data of the depth camera is processed, the camera intrinsic parameters are used to extract the image depth, and the three-dimensional point cloud data set of the back of the detected person is constructed.

[0021] The difference between the human body and the room temperature is used to set a threshold, and all point clouds in the temperature image data that exceed the threshold are extracted to obtain the human thermal image.

[0022] The body posture data is dynamically fused using complementary filtering to obtain the posture angle of the current body posture in the three-dimensional space; and the deviation angle is calculated according to the included angle between the acceleration vector and the standard gravity direction.

[0023] For the plantar pressure distribution map, the plantar pressure distribution map is symmetrically divided into two parts, the effective contact area of each part is calculated to obtain the support area and the distribution map.

[0024] As a preferred implementation, the reconstructing the back three-dimensional curved surface model using the Poisson surface reconstruction algorithm specifically comprises:

[0025] According to the distance of the camera from the detection point, a depth range is set, the effective point set of the detected person's back is extracted, and statistical filtering is used to remove outliers in the effective point set.

[0026] Based on the filtered effective point set, the normal vector is calculated, and then the depth camera position is taken as a reference viewpoint, and the normal vector direction is adjusted to be consistent with the viewpoint.

[0027] An adaptive octree space structure is constructed, and a cube surrounding space covering the whole point cloud range is initialized according to the maximum boundary length of the whole point cloud data;

[0028] The cube is segmented, and the segmentation depth is dynamically adjusted according to the point cloud density;

[0029] A corresponding third-order B-spline basis function is constructed for each segmented node, and a corresponding gradient vector field is calculated by numerical differentiation according to the connection relationship between the nodes;

[0030] For each sampling point of the input point cloud, a plurality of basis functions are selected in the neighborhood of the sampling point, and a weight coefficient is assigned to the sampling point according to the spatial position relationship between the sampling point and each basis function center by using a Gaussian weight. The approximate normal vector at the sampling point is obtained by weighting and linearly combining all the basis function gradient vectors in the neighborhood of the sampling point. The process is executed point by point on the whole point cloud to construct a complete normal vector field;

[0031] Based on the estimated normal vector field, a discrete system of Poisson equation is constructed, the B-spline basis function gradient is assembled by using a sparse matrix structure, a zero normal flux boundary condition is introduced to constrain the boundary deformation, and a zero mean constraint is added to eliminate the constant drift of the solution;

[0032] The conjugate gradient method is used to iteratively solve the linear system to obtain a scalar potential function defined in a three-dimensional space;

[0033] Based on the moving cube algorithm, an isosurface is extracted from the potential function to generate a three-dimensional surface model of the back.

[0034] As a preferred embodiment, the posture correction of the three-dimensional surface model of the back specifically includes:

[0035] The body posture data output by the IMU sensor is converted into the reconstruction coordinate system in which the three-dimensional model of the back is located;

[0036] The obtained posture angle is used as a whole correction basis, and the model is corrected by inverse transformation around the coordinate axis;

[0037] The obtained deviation angle is used as a global constraint to perform posture normalization processing on the three-dimensional surface model;

[0038] According to the left and right contact areas of the foot bottom, the geometric center of gravity point of the foot bottom is calculated by ignoring the specific gravity data;

[0039] The Euclidean distance between the actual center of gravity point and the geometric center of gravity point is calculated to determine the deviation;

[0040] The difference between the actual barycenter point and the geometric barycenter point is converted into forward and backward and left and right components, and the back three-dimensional surface model is corrected in reverse according to the difference components.

[0041] As a preferred embodiment, the back thermal correction model specifically comprises:

[0042] By joint calibration and time synchronization of the internal and external parameters of the depth camera and the infrared device, the coordinate mapping relationship between the thermal map and the back three-dimensional surface model is established, and the temperature data of the thermal map is projected and interpolated to the vertex of the back three-dimensional surface model.

[0043] A temperature-driven normal deformation is defined at the vertex of the model corresponding to the abnormal area, and a Laplace smoothing algorithm based on thermal weight is used for iterative optimization to generate the back thermal correction model.

[0044] As a preferred embodiment, the extraction of the spinal segment markers by the landmark point recognition algorithm specifically comprises:

[0045] The coordinates of the spinal segment markers of the top and bottom of the spine in the three-dimensional space are recognized by the landmark point recognition algorithm.

[0046] The landmark point recognition algorithm directly processes three-dimensional point cloud data through the PointNet network architecture, independently encodes each point feature using a shared multi-layer perception machine, realizes global feature aggregation through maximum pooling to overcome the unstructured characteristics of point cloud, and pre-trains through a point cloud data set containing landmark points to obtain a converged model to recognize the part containing landmark points in the point cloud data.

[0047] According to the extracted landmark points, a segmented cubic B-spline curve is used to fit the spinal midline to obtain the spinal midline shape of the back three-dimensional surface model, and the center points of each spinal segment marker are marked.

[0048] As a preferred embodiment, the thermal correction of the spinal midline specifically comprises:

[0049] Abnormal areas with a local high-low temperature difference exceeding 1.5℃ are extracted, and a deformation displacement along the geometric normal vector of the corresponding vertex is applied at the corresponding vertex.

[0050] The position is optimized by using thermal weight Laplace smoothing, the segmented cubic B-spline curve is re-solved based on the corrected control point set, and the spinal midline fitting the thermal condition is obtained.

[0051] As a preferred embodiment, the judgment of the scoliosis condition and the output of the report specifically comprises:

[0052] According to the mark points marked on the midline of the spine, the upper end cone and the lower end cone of the spine are distinguished, and the inclination of each mark point is calculated for the upper end cone and the lower end cone respectively;

[0053] For the mark points with the maximum inclination of the upper end cone and the lower end cone, Cobb angle calculation is performed, and the adjacent two mark points of each are used as auxiliary to construct the upper end cone inclination direction vector and the lower end cone inclination direction vector, and then the Cobb angle of the spine in the section is obtained by using the included angle between the two vectors.

[0054] According to the Cobb angle, the scoliosis condition is judged, the analysis process is integrated, and a report including an analysis model picture and a scoliosis condition is output.

[0055] The beneficial effects of the present application are:

[0056] 1. Improve detection safety: Based on the non-radiation detection method of three-dimensional imaging, the ionizing radiation risk caused by X-ray imaging is avoided, and it is more suitable for repeated detection of adolescent population.

[0057] 2. Enhance the stability and accuracy of posture analysis: By fusing posture angle, support force distribution and heat map and other multi-source data, the body posture of the detected person is dynamically corrected, and the interference of human posture control on the detection result is significantly reduced.

[0058] 3. Restore the real shape of the spine and quantitatively evaluate the results: Through point cloud reconstruction and heat guided model deformation correction, combined with automatically recognized vertebral landmark points and midline fitting algorithm, a three-dimensional model of the spine is constructed, Cobb angle is automatically calculated, and objective quantitative analysis of the degree of scoliosis is realized, which is convenient for early diagnosis and intervention in clinic.

[0059] Legend

[0060] Figure 1 A flowchart of a juvenile scoliosis detection method based on three-dimensional imaging in the present application. DETAILED DESCRIPTION

[0061] In order to make the technical means, creative features, purposes and effects realized by the present application easy to understand, the present application will be further described below in combination with specific embodiments, but the following embodiments are only preferred embodiments of the present application, not all. Based on the embodiments in the embodiments, other embodiments obtained by those skilled in the art without creative labor also belong to the protection scope of the present application. Unless otherwise specified, the experimental methods in the following embodiments are all conventional methods, and the materials, reagents and the like used in the following embodiments can be obtained from commercial channels unless otherwise specified.

[0062] Example 1 is as follows: Figure 1A flow chart of a three-dimensional imaging-based adolescent scoliosis detection method is shown, which embodiment provides a three-dimensional imaging-based adolescent scoliosis detection method, specifically comprising the following steps:

[0063] Step one: obtain multi-modal data about the spine of the detected person through various sensors, and upload the obtained data to the cloud platform:

[0064] The specific steps are: setting various types of sensors at the detection point, guiding the detected person to perform data collection work in a standard posture, and uploading the obtained data to the cloud platform through a wireless network.

[0065] Specifically, setting various types of sensors at the detection point means that the sensor types include depth cameras, infrared imaging devices, IMU sensors (inertial measurement units), and plantar pressure distribution sensors, the depth camera is used to obtain the depth map of the back of the detected person, the infrared imaging device is used to obtain the temperature data of the back of the detected person, the IMU sensor is used to obtain the body posture data of the detected person, including three-axis attitude angle and dynamic stability information, and the plantar pressure distribution sensor is used to obtain the plantar pressure data of the detected person.

[0066] Guiding the detected person to perform data collection work in a standard posture means that the room temperature is lower than 28℃, the detected person is guided to stand on the plantar pressure distribution sensor in the detection area with parallel feet; facing away from the depth camera and the infrared imaging device, the depth camera and the infrared imaging device can clearly obtain the complete image and thermal data of the back of the detected person; the IMU sensor is worn on the middle line of the back of the detected person near the seventh cervical vertebra to the first thoracic vertebra region to obtain the body posture data of the detected person.

[0067] Step two: after the cloud platform receives the original data, the original data is preprocessed to obtain standardized data, and then the standardized data is analyzed to obtain modeling data.

[0068] Specifically, the preprocessing of the original data refers to: using an optical correction algorithm based on an intrinsic matrix to correct image distortion for complete image data collected by a depth camera and thermal images collected by an infrared thermal imaging device; performing proportion unification and edge cropping on the images to eliminate background interference; applying a Gaussian filter image denoising algorithm to reduce photosensitive device noise interference; applying a Kalman filter algorithm to the original angular velocity and acceleration data in the body posture data collected by the IMU sensor, establishing a state space model containing angle, angular velocity, acceleration bias, etc., predicting the system state at the next time using the state equation and calculating the prediction covariance, combining the current sensor measurement value to calculate the Kalman gain, adjusting the weight ratio of the predicted value and the measured value based on the Kalman gain, and generating the optimal state estimation through weighted fusion, thereby effectively filtering out high-frequency noise and real-time compensating for sensor zero drift error while preserving the true motion trend; and removing the starting redundant data during the collection process, unifying the output dimensions and formats; performing threshold elimination on the low-value noise points in the plantar pressure matrix in the plantar pressure distribution data to reduce non-contact area interference, and unifying the left and right foot pressure maps to the standard template coordinate system to provide a basis for subsequent matching or analysis.

[0069] The analysis of the standardized data refers to: first, processing the depth map image data of the depth camera to construct a three-dimensional point cloud data set of the back of the detected person. Each pixel of the image data output by the depth camera contains a depth value, i.e., the distance from the camera. The depth camera itself also contains an imaging parameter, including focal length and principal point coordinates. According to the coordinate position of the pixel in the image, combined with the depth value, the three-dimensional coordinate value of each pixel is formed, thereby forming a complete three-dimensional point cloud data set for modeling to represent the back surface structure of the detected person.

[0070] Subsequently, for the back temperature image data obtained by the infrared imaging device, the difference between the human body and the room temperature is utilized, 32°C is set as the threshold value, the values greater than 32°C in the temperature image data are extracted, and the back thermal image is obtained.

[0071] Then, for the body posture data obtained by the IUM sensor, including acceleration and angular velocity, the acceleration + angular velocity data is dynamically fused using a complementary filter to obtain the posture angle of the current body posture in the three-dimensional space; then the deviation angle is calculated. In the state of the detected person standing still, the acceleration value collected by the IUM sensor is mainly composed of gravitational acceleration, so this acceleration vector can be used as an approximate reflection of the trunk gravity direction. The direction of this acceleration vector in the sensor coordinate system is calculated with the standard gravity direction (i.e., the vertical downward direction) to obtain the spatial included angle.

[0072] Finally, for the plantar pressure distribution data, the left-right support force difference is calculated. Specifically, when the human body is standing still, the contact pressure distribution between the feet and the ground can be regarded as a scalar field of discrete sampling points, each sampling point reflecting the normal pressure value at that position. According to the principle of mechanics, the total support force of a single foot is equal to the integral of the pressure values of all contact points and the corresponding areas. According to the standard posture, the plantar pressure distribution data is divided into two parts symmetrically on the left and right sides. The left-right support force difference is calculated by calculating the integral of the corresponding areas. The support area and distribution map are calculated. The foot support area reflects the actual contact range between the foot and the ground. If the contact area is insufficient or there is a significant difference between the left and right sides, it is usually related to the habit of standing posture, abnormal foot posture, etc. In the discrete sampled pressure field, when the pressure value of a certain point exceeds the minimum contact threshold, it is considered as an effective contact point. The specific minimum contact threshold is set according to the weight of the detected person. The average unit pressure can be obtained by dividing the weight by the average plantar contact area of the detected person with the same weight, and then multiplied by the empirical coefficient of similar weight as the threshold. Through the effective contact point statistics, the support area and distribution map are obtained.

[0073] Step three: build a virtual model of the back of the detected person according to the modeling data, modify the virtual model of the back in combination with other data, and generate a back thermal correction model in combination with the back thermal map:

[0074] The specific steps are: according to the three-dimensional point cloud data set, a back model of the detected person is constructed; the analysis posture angle, deviation angle, left-right support force difference, and foot support area and distribution map are loaded to modify the back model; according to the back model image, a back thermal distribution model is constructed in combination with the human body thermal map.

[0075] Specifically, constructing a back model of the detected person means that in the three-dimensional point cloud data set of the back image, each point has a depth value. According to the distance between the camera and the detection point, a depth range is set. The depth range is adjusted according to the actual distance to ensure that the back condition of the detected person can be retained. According to the depth range, the three-dimensional point cloud data set is filtered, and the points within the range are retained as the effective point set. Then, the effective point set is further processed. A neighborhood of 15 neighboring points is constructed for each point using statistical filtering. The distance average of each point in its neighborhood and other points is calculated, and then the distance average distribution of all points is calculated. If the distance average of a certain point exceeds the overall average plus 1.5 times the standard deviation, the point is considered as an outlier and is removed, thereby retaining the effective point set with dense distribution and reasonable structure.

[0076] After the completion of the effective point set of outlier removal, the Poisson surface reconstruction algorithm is used to accurately reconstruct the three-dimensional surface model of the back of the detected person. The specific process is as follows: first, based on the screened effective point set, the normal vector is calculated, and 15 adjacent points in the neighborhood of each point are taken. The minimum eigenvalue corresponding to the eigenvector is determined by eigenvalue decomposition of the covariance matrix as the initial normal direction; then the depth camera position is taken as the reference view point, the normal vector direction is adjusted to be consistent with the view point, the point being processed is connected with the view point, and the angle between the normal vector and the line segment is calculated. If the angle is greater than 90°, it means that the normal vector points in the opposite direction, and it is inverted to avoid the direction ambiguity in surface reconstruction; then an adaptive octree spatial structure is constructed, and the maximum boundary length of the whole point cloud data is taken as the scale to initialize the generation of a cubic enclosing space covering the whole point cloud range as the root node of the octree; then the cubic is segmented, and the segmentation depth is dynamically adjusted according to the point cloud density. The initial maximum depth is set to 8 to ensure that the high-density area is expressed finely and the low-density area maintains reasonable topology. The specific setting can be modified according to the actual situation; in the three-dimensional space divided by the octree, for each leaf node obtained by subdivision, a local supported three-order B-spline basis function is constructed in its corresponding space. The basis function has non-zero value only in the current node and its adjacent nodes to ensure the efficiency and numerical stability of local calculation; according to the pre-defined connection relationship of the octree nodes (such as the topological relationship between parent and child nodes and adjacent nodes), the gradient of each basis function is derived. Specifically, the gradient components of the basis function in X, Y and Z directions in three-dimensional space are calculated by numerical differentiation, and then the corresponding gradient vector field is obtained; then, for each sampling point of the input point cloud, a plurality of basis functions are selected in its neighborhood, and a weight coefficient is assigned according to the spatial position relationship between the point and each basis function center by using Gaussian weight, so as to ensure that the basis function closer to the point has greater influence on the point. The weighted linear combination of the gradient vectors of all basis functions in the neighborhood of the current processing point is carried out to obtain the approximate normal vector at the sampling point. This process is executed point by point on the whole point cloud to construct a complete normal vector field; based on the estimated normal vector field, a discrete system of Poisson equation is constructed, the B-spline basis function gradient is assembled by using sparse matrix structure, the zero normal flux boundary condition is introduced to constrain the boundary deformation, and the zero mean constraint is superimposed to eliminate the constant drift of the solution, so as to ensure the uniqueness and stability of the solution of Poisson equation. The zero normal flux boundary condition means that in the boundary region of the point cloud, the component of the corresponding basis function gradient in the boundary normal direction is set to zero, which means that the flow field does not penetrate at the boundary.This is by adding a linear constraint of the normal component of the boundary sampling point being 0; the zero mean constraint refers to: adding an additional condition in the overall solution space, so that the average value of all estimated normals is zero, and the normal vector sum of all points is set to the zero vector, as a linear constraint equation added to the optimization objective; the conjugate gradient method is used to iteratively solve the linear Poisson system constructed, to obtain the scalar potential function field defined in the three-dimensional space; then, based on the marching cubes algorithm, a specific isosurface is extracted from the potential function, and the isosurface threshold is determined by automatically calculating the median of the potential function values corresponding to all point clouds, so as to ensure that the extracted isosurface is closest to the original data distribution, and finally a smooth, closed and topologically reasonable back three-dimensional surface model is generated.

[0077] The posture angle, deviation angle, left-right support force difference, foot support area and distribution map are loaded to correct the back model. First, the roll angle, pitch angle and yaw angle data output by the IMU sensor are converted to the reconstruction coordinate system of the back three-dimensional model. The obtained posture angle is used as the basis for overall correction. The overall rotation, inclination or distortion caused by the deviation of the standing posture during detection is corrected by inverse transformation around the coordinate axis, so as to adjust the model posture to the standard reference posture. The obtained deviation angle is used as a global constraint to perform posture normalization processing on the three-dimensional curved surface model, that is, the model is subjected to reverse rotation operation around the coordinate axis, and the overall inclination, rotation and other posture distortion introduced by the body position deviation of the detected person are gradually eliminated, so as to ensure that the model form is consistent with the true back structure under the natural standing body position. Subsequently, according to the position of the left and right contact areas and the pressure center of the foot, the actual center of gravity point is calculated. The actual center of gravity point can be calculated by weighted average of the pressure values and corresponding position coordinates of each measuring point in the foot pressure distribution map. The pressure values of each foot area are multiplied by their coordinate positions in the horizontal and vertical directions, and the total horizontal and vertical moments of all areas are obtained, and then divided by the total pressure, so as to obtain the horizontal and vertical coordinates of the center of gravity on the plane, so as to determine the average projection position of the gravity of the human body on the ground under the static standing state, that is, the actual center of gravity point. Then, according to the contact area map, the geometric center of gravity point is obtained. Each point in the contact area is regarded as a pixel unit with uniform mass, and their horizontal and vertical coordinate values are counted respectively. Then, the horizontal coordinates of all these points are averaged to obtain the center position of the contact area in the horizontal direction, and the vertical coordinates are averaged to obtain the center position in the vertical direction. Finally, the average coordinates in the two directions are combined to obtain the geometric center of gravity point of the entire foot contour area. Then, the Euclidean distance between the actual center of gravity point and the geometric center of gravity point is calculated. When it exceeds the set threshold value, the correction work is triggered. The threshold value is initially set to 0.7 cm and can be changed according to the actual weight of the detected person. Then, the difference is converted into a correction amount. Specifically, the difference is decomposed into forward and backward components and left and right components, which correspond to the posture deviation of the sagittal plane and the coronal plane respectively. According to the deviation, the back three-dimensional curved surface model is corrected in the reverse direction to restore the spinal morphology of the detected person under natural conditions. Thus, the back three-dimensional curved surface model basically excluding human active control is obtained.

[0078] The back thermal force correction model combined with the human thermal map is to establish the coordinate mapping relationship between the thermal map and the back three-dimensional surface model: through the joint calibration of the internal and external parameters of the depth camera and the infrared equipment and time synchronization, the emissivity and radiation calibration of the thermal image data are completed, the target that can be recognized by the two devices at the same time is used for cross-modal registration, the temperature data of the thermal map is projected and interpolated to the vertex of the back three-dimensional surface model; for the occluded and out-of-field area, the visibility is judged and the hollow area is filled; based on temperature anomaly detection to determine the area that needs to be deformed: through the left and right symmetric ROI temperature difference along the spine line as the basis for judgment, whether the current temperature deviates from the mean value of the temperature difference is used for judgment, and 1.5℃ is initially used as the floating upper and lower limit to identify the abnormal temperature area; for the detected abnormal area, define the temperature-driven normal deformation at the corresponding model vertex; the displacement amount of the normal deformation is proportional to the deviation of the temperature of the vertex from the back temperature reference (taking the mean value of the mapping area); next, the Laplace smoothing algorithm based on thermal force weight is used for iterative optimization, and greater smoothing weight is given to the area with smaller temperature difference to suppress noise; the smoothing is weakened in the feature area with significant temperature difference to retain the details of the deformation, and the iterative convergence threshold is set to ensure the stability of the optimization process. Finally, the generated back thermal force correction model not only retains the consistency of the back bony structure, but also shows the asymmetric change of muscle tension through the thermal map.

[0079] Step four: combine the generated virtual model, locate the spine midline of the detected person, correct the shape of the spine midline combined with the thermal force correction model, analyze the spine scoliosis of the detected person according to the spine midline, and output the spine scoliosis report.

[0080] Specifically, the analysis of scoliosis refers to: first, positioning the spinal midline according to the three-dimensional surface model of the back; obtaining the coordinates of the spinal segment markers (such as C7 and L5 / S1) at the top and bottom of the spine in three-dimensional space, identifying these key marker points through a marker point recognition algorithm; the marker point recognition algorithm directly processes three-dimensional point cloud data through the PointNet network architecture, independently encodes the features of each point using shared multi-layer perceptron, and realizes the global feature aggregation of permutation invariance through maximum pooling to overcome the unstructured characteristics of point cloud; based on this deep learning method, an end-to-end training process is constructed: first, prepare a dataset containing three-dimensional coordinates of spinal segment markers C7 / L5 / S1 and corresponding point clouds, take the point coordinates and normal vectors as network inputs, output the predicted position after forward propagation, calculate the deviation of the predicted value from the true coordinates using the smooth L1 loss function, and constrain the spatial vector consistency combined with the direction cosine similarity; dynamically optimize the network parameters through back propagation until convergence; input the back point cloud data of the detected person into the PointNet model to generate the marker point coordinates; after obtaining the coordinates of the spinal segment markers C7 and L5 / S1, identify the key points of the vertebral bodies such as T6 / T12 / L3 as control nodes in the same way, fit the spinal midline using a segmented cubic B-spline curve, and obtain the spinal midline morphology of the three-dimensional surface model of the back, wherein the center points of each spinal segment marker are marked as marker points.

[0081] Next, based on the generated spinal midline and the back thermal correction model, the midline geometry correction is driven by the temperature distribution of the heat map and the vertex mapping relationship of the three-dimensional surface model: extract the abnormal area with a high-low temperature difference exceeding 1.5°C (usually corresponding to the muscle tension imbalance zone), apply a deformation displacement along the geometric normal vector of the corresponding vertex at the corresponding vertex, and the displacement amount in the deformation displacement is proportional to the temperature difference, and the proportional coefficient is initially preset as 0.2mm / °C, and the upper limit constraint is about 3mm; for the vertebral body control points affected by thermal deformation, position optimization is performed using thermal weight Laplace smoothing, a strong smoothing weight is applied in the flat area with a temperature difference less than 0.5° to maintain the stability of the bone structure, and a weak smoothing weight is used in the abnormal zone with a temperature difference greater than 1.5° to retain the deformation characteristics; finally, based on the corrected control point set, the segmented cubic B-spline curve is solved again to analyze and obtain the spinal midline fitting thermal conditions.

[0082] Finally, according to the obtained spine midline, the scoliosis condition is analyzed, and according to the marked points marked on the spine midline, the upper end cone and the lower end cone of the spine are distinguished, specifically, L1-L3 is the upper end cone, and the other part is the lower end cone; for the upper and lower end cones, the inclination of each marked point is calculated, specifically, the first derivative is calculated at each marked point according to the cubic B-spline curve, the tangent vector direction is obtained, the included angle between the tangent vector and the vertical direction unit vector is calculated, and the marked point with the maximum included angle is recorded as the maximum inclination; for the marked point with the maximum inclination of the upper and lower end cones, the Cobb angle is calculated, and the respective adjacent two marked points are used as auxiliary to construct the upper end cone inclination direction vector and the lower end cone inclination direction vector, then the included angle between the two vectors is used as the basis for Cobb angle calculation, that is, the included angle value between the two direction vectors is calculated through the inverse cosine function, so as to obtain the curvature degree of the spine in the scoliosis section, and to judge whether there is a scoliosis problem, if the angle exceeds the clinically defined 10° threshold, it is determined as scoliosis, and the severity is evaluated according to the angle size, Cobb angle 10°-25° is mild, 25°-45° is moderate, and more than 45° is severe, after obtaining the result, the report including the analysis model picture and the scoliosis condition is output.

[0083] Example 2:

[0084] In order to perform early screening and dynamic review of adolescent scoliosis, a hospital deploys the adolescent scoliosis detection system based on three-dimensional imaging as described in the present application, and the room temperature is maintained at about 22°C, the illumination is uniform, and there are no strong shadows or reflective areas.

[0085] Before detection, medical staff first fix a plantar pressure distribution sensing mat on the ground in the detection area, install a depth camera and an infrared thermal imaging device about 2 meters away from the ground on one side, keep the relative position unchanged through a fixed support, and complete joint calibration, and the IMU sensor is worn on the back of the detected adolescent between the seventh cervical vertebra and the first thoracic vertebra after disinfection, and is fixed and adhered to prevent slipping.

[0086] The detected person stands in the center of the plantar pressure distribution sensing mat according to the prompt, spreads the feet parallelly to the same width as the shoulders, and faces away from the camera and the thermal imaging device, raw body data is collected, and then the raw data is uploaded to the cloud platform through a wireless network:

[0087] On the cloud platform, the received original data is first preprocessed; then, the cloud platform further analyzes to obtain a three-dimensional point cloud data set, and a smooth and closed three-dimensional surface model of the back is obtained through a Poisson surface reconstruction algorithm; combined with the IMU attitude angle data and the plantar pressure distribution data analysis, it is found that the detected person has a certain center of gravity deviation due to the light weight, which may cause the identification result of the spinal segment to be inaccurate, have errors, and cause the scoliosis condition to be covered up; therefore, the model is modified according to the center of gravity deviation, and the error of the spinal detection result caused by the unstable standing or incorrect posture of the detected person is reduced.

[0088] Subsequently, the infrared thermal map is projected to the vertex of the three-dimensional surface model, a normal deformation driven by heat is applied to the local area according to the left-right symmetric temperature difference distribution, and a Laplace smoothing optimization of the heat weight is performed to form a back thermal correction model.

[0089] After the model is established, a landmark recognition algorithm is called to recognize the landmark points of the spinal segments of the model, the C7, L5 / S1 and T6 / T12 / L3 key vertebral coordinate points are extracted, and a segmented cubic B-spline curve fitting is used to fit the spinal midline; for the local area with a high-low temperature difference of more than 1.5 DEG C, a geometric correction of the midline is performed to obtain the finally fitted spinal midline.

[0090] Finally, the maximum inclination point of the upper and lower end vertebrae is calculated, the Cobb angle is calculated according to the maximum inclination point of the upper and lower end vertebrae, and a detection report is generated; the report shows that the calculation result shows that the Cobb angle of the spine of a detected person is 13.7 DEG, and it is determined to be a mild scoliosis; the report also includes a three-dimensional back model diagram, a spinal midline shape diagram and a mechanical and thermal distribution analysis diagram, which are used for doctors to develop subsequent rehabilitation training programs and review plans.

[0091] The above shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above-mentioned embodiments, and the above-mentioned embodiments and descriptions in the specification are only preferred examples of the present application and are not intended to limit the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for detecting scoliosis of adolescents based on three-dimensional imaging, characterized by: setting up various sensors to obtain body data of the detected person, guiding the detected person to collect data in a standard posture, obtaining raw data, and uploading the raw data obtained by the sensors to a cloud platform; the cloud platform pre-processes the raw data obtained by the sensors into standardized data; analyzes the standardized data to obtain three-dimensional point cloud data sets, posture correction data and pressure correction data for modeling; further effective point screening is performed according to the three-dimensional point cloud data sets, and a back three-dimensional surface model is reconstructed by using a Poisson surface reconstruction algorithm; combined with the posture correction data, the back three-dimensional surface model is corrected in posture; a human body thermal map is introduced to construct a back thermal correction model; according to the back three-dimensional surface model, a landmark recognition algorithm is used to extract the landmarks of the spinal segments to obtain the morphology of the spinal midline; based on the generated spinal midline and the back thermal correction model, the thermal correction of the spinal midline is performed; the spinal midline is divided into upper and lower vertebral bodies, the Cobb angle is calculated, the scoliosis condition is judged, and a report is output.

2. The method for detecting scoliosis in adolescents based on three-dimensional imaging according to claim 1, characterized in that: The raw data obtained specifically includes: setting up various types of sensors at the detection point, the sensor types including depth camera, infrared imaging device, IMU sensor and foot pressure distribution sensor, obtaining the back depth map of the detected person by the depth camera, obtaining the back temperature data of the detected person by the infrared imaging device, obtaining the body posture data of the detected person by the IMU sensor, and obtaining the foot pressure data of the detected person by the foot pressure distribution sensor; guiding the detected person to collect data in a standard posture, the specific requirements being: the room temperature is lower than 28℃, guiding the detected person to stand on the foot pressure distribution sensor in the detection area, the two feet are parallel and apart, the back is against the depth camera and the infrared imaging device, and the IMU sensor is worn on the detected person's back midline near the seventh cervical vertebra to the first thoracic vertebra region; uploading the obtained raw data to the cloud platform through wireless network.

3. The method for detecting scoliosis in adolescents based on three-dimensional imaging according to claim 1, characterized in that: The conversion into standardized data specifically includes: applying an optical correction algorithm based on an intrinsic matrix to correct the image distortion of the complete image data collected by the depth camera and the thermal map collected by the infrared thermal imaging device, uniformly scaling the image and edge cropping, eliminating background interference, and applying a Gaussian filter image noise reduction algorithm to reduce the noise interference of the photosensitive device; applying Kalman filter algorithm to the original angular velocity and acceleration data in the body posture data collected by the IMU sensor to reduce high-frequency jitter and zero-point drift, and removing the redundant data in the collection process to unify the output dimension and format; thresholding the low-value noise points in the foot pressure matrix in the foot pressure distribution data to reduce non-contact area interference, and unifying the left and right foot pressure maps to the standard template coordinate system.

4. The method for detecting scoliosis in adolescents based on three-dimensional imaging according to claim 1, characterized in that: The analysis of the standardized data specifically includes: processing the depth map image data of the depth camera, extracting the image depth by using the camera intrinsic parameters, and constructing the three-dimensional point cloud data set of the back of the detected person; The human body thermal image is obtained by setting a threshold, using the difference between the human body and the room temperature, extracting all point clouds exceeding the threshold from the temperature image data, and obtaining the human body thermal image; The body posture data is dynamically fused by using complementary filtering to obtain the posture angle of the current body posture in the three-dimensional space; and the deviation angle is calculated according to the included angle between the acceleration vector and the standard gravity direction. The plantar pressure distribution map is symmetrically divided into two parts, the effective contact area of each part is calculated, and the support area and the distribution map are obtained.

5. The method for detecting scoliosis in adolescents based on three-dimensional imaging according to claim 1, characterized in that: The back three-dimensional surface model is reconstructed by using the Poisson surface reconstruction algorithm, and the back three-dimensional surface model is reconstructed by using the Poisson surface reconstruction algorithm. According to the distance between the camera and the detection point, the depth range is set, the effective point set of the back of the detected person is extracted, and the statistical filtering is used to remove the outliers in the effective point set; Based on the filtered effective point set, the normal vector is calculated, and then the depth camera position is taken as the reference view point, and the normal vector direction is adjusted to be consistent with the view point; An adaptive octree space structure is constructed, and the maximum boundary length of the whole point cloud data is taken as the scale to initialize the generation of a cubic enclosing space covering the whole point cloud range; The cubic is segmented, and the segmentation depth is dynamically adjusted according to the point cloud density; For each segmented node, the corresponding third-order B-spline basis function is constructed, the corresponding gradient vector field is calculated by using the numerical differentiation calculation through the connection relationship between the nodes; For each sampling point of the input point cloud, a plurality of basis functions are selected in the neighborhood, a weight coefficient is assigned to the space position relationship between the point and each basis function center by using the Gaussian weight, and the approximate normal vector at the sampling point is obtained by performing a weighted linear combination on all basis function gradient vectors in the neighborhood of the point being processed. Based on the estimated normal vector field, a discrete system of Poisson equation is constructed, the B-spline basis function gradient is assembled by using a sparse matrix structure, a zero normal flux boundary condition is introduced to constrain the boundary deformation, and a zero mean constraint is superimposed to eliminate the constant drift of the solution. The conjugate gradient method is used to iteratively solve the linear system to obtain the scalar potential function defined in the three-dimensional space; Based on the moving cubic algorithm, the isosurface is extracted from the potential function to generate the back three-dimensional surface model.

6. The method for detecting scoliosis in adolescents based on three-dimensional imaging according to claim 1, characterized in that: The posture of the back three-dimensional surface model is corrected, and the posture of the back three-dimensional surface model is corrected. The body posture data output by the IMU sensor is converted into the reconstruction coordinate system of the back three-dimensional model; The obtained posture angle is used as the basis for overall correction, and the model is corrected by inverse transformation around the coordinate axis; The deviation angle is used as a global constraint to perform posture normalization processing on the three-dimensional surface model; The geometric gravity point of the foot is calculated according to the left and right contact areas of the foot, and the specific gravity data is ignored. The Euclidean distance between the actual gravity point and the geometric gravity point is calculated to determine the deviation. The difference between the actual gravity point and the geometric gravity point is converted into forward and backward components and left and right components, and the back three-dimensional surface model is corrected in the reverse direction according to the difference components.

7. The method for detecting scoliosis in adolescents based on three-dimensional imaging according to claim 1, characterized in that: The back thermal correction model is constructed, and the back thermal correction model is constructed. The coordinate mapping relationship between the thermal map and the back three-dimensional surface model is established by joint calibration and time synchronization of the internal and external parameters of the depth camera and the infrared device, and the temperature data of the thermal map is projected and interpolated to the vertex of the back three-dimensional surface model; A normal deformation driven by temperature is defined at the vertex of the model corresponding to the abnormal area, and a Laplace smoothing algorithm based on thermal weight is used for iterative optimization to generate a back thermal correction model.

8. The method for detecting scoliosis in adolescents based on three-dimensional imaging according to claim 1, characterized in that: The specific steps of the landmark point recognition algorithm for extracting the spinal segment landmarks include: The coordinates of the spinal segment landmarks at the top and bottom of the spine in the three-dimensional space are recognized by the landmark point recognition algorithm; The landmark point recognition algorithm directly processes three-dimensional point cloud data through the PointNet network architecture, independently encodes each point feature by using a shared multi-layer perception machine, realizes global feature aggregation with permutation invariance through maximum pooling, overcomes the unstructured characteristics of point cloud, and pre-trains through a point cloud data set containing landmark points to obtain a converged model to recognize the part containing landmark points in the point cloud data; According to the extracted landmark points, a segmented cubic B-spline curve is used to fit the spinal midline to obtain the shape of the spinal midline of the back three-dimensional surface model, and the center points of the spinal segment landmarks are marked.

9. The method for detecting scoliosis in adolescents based on three-dimensional imaging according to claim 1, characterized in that: The specific steps of the thermal correction of the spinal midline include: Abnormal areas with a local high-low temperature difference of more than 1.5℃ are extracted, and a deformation displacement along the geometric normal vector of the corresponding vertex is applied at the corresponding vertex; The position is optimized by using thermal weight Laplace smoothing, the segmented cubic B-spline curve is re-solved based on the corrected control point set, and the spinal midline fitting the thermal condition is obtained.

10. The method for detecting scoliosis in adolescents based on three-dimensional imaging according to claim 1, characterized in that: The specific steps of judging the scoliosis condition and outputting the report include: According to the marker points marked on the spinal midline, the upper and lower end cones of the spine are distinguished, and the inclination of each marker point is calculated for the upper and lower end cones. For the marker points with the largest inclination of the upper and lower end cones, the Cobb angle is calculated, the adjacent two marker points of each are used as auxiliary, the upper end vertebrae inclination direction vector and the lower end vertebrae inclination direction vector are constructed, and then the Cobb angle of the spine in the segment is obtained by using the included angle between the two vectors. According to the Cobb angle, the scoliosis condition is judged, the analysis process is integrated, and a report including the analysis model picture and the scoliosis condition is output.

Images