A spinal three-dimensional posture detection system

Abstract

The present application relates to a kind of spine three-dimensional posture detection system, and it is related to scoliosis diagnostic technical field.The system mainly includes: through the gait video sequence of the video acquisition module multi-angle acquisition of the detected person in natural walking state;First detection module determines the scoliosis risk level of the detected person according to gait video sequence;If scoliosis risk level is greater than preset level, then through multimodal data acquisition module, the X-ray film of the full spine of the detected person in standard static posture, millimeter wave imaging data, infrared thermal imaging data and plantar pressure data are synchronously collected;Second detection module generates scoliosis diagnosis report according to the X-ray film of the full spine of the detected person in standard static posture, millimeter wave imaging data, infrared thermal imaging data and plantar pressure data;Diagnosis output module outputs the scoliosis diagnosis report corresponding to the detected person.

Description

Technical Field

[0001] This invention relates to the field of scoliosis diagnosis technology, and in particular to a three-dimensional posture detection system for the spine. Background Technology

[0002] Scoliosis, commonly known as "spinal curvature," is not merely a "C" or "S" shaped curve of the spine in the left-right direction; it is a three-dimensional deformity. This means that in addition to lateral curvature, it is usually accompanied by vertebral rotation and changes in the physiological curvature in the sagittal plane.

[0003] Currently, a Cobb angle greater than 10 degrees, measured professionally on a standing full-spine X-ray, is sufficient to diagnose scoliosis. However, the Cobb angle is a single, static measurement on a static standing X-ray. The human body is in dynamic activity (walking, bending, prolonged sitting), and a static angle cannot reflect the stability of the spine under dynamic loads, muscle compensation capacity, or fatigue levels. This results in the relatively low accuracy of current scoliosis detection methods. Summary of the Invention

[0004] The present invention aims to provide a three-dimensional posture detection system for the spine to overcome the shortcomings of the existing technology. The technical problem to be solved by the present invention is achieved through the following technical solution.

[0005] This invention provides a three-dimensional spinal posture detection system, which includes: a video acquisition module, a multimodal data acquisition module, a first detection module, a second detection module, and a diagnostic output module; The video acquisition module acquires gait video sequences of the subject in a natural walking state from multiple angles. The first detection module determines the scoliosis risk level of the subject based on the gait video sequence; If the risk level of scoliosis is greater than the preset level, the multimodal data acquisition module will simultaneously acquire X-ray images, millimeter-wave imaging data, infrared thermal imaging data and plantar pressure data of the whole spine of the subject in a standard static posture. The second detection module generates a scoliosis diagnosis report based on the X-ray of the entire spine, millimeter-wave imaging data, infrared thermal imaging data, and plantar pressure data of the subject in a standard static posture. The diagnostic output module outputs a scoliosis diagnostic report for the person being tested.

[0006] In an optional embodiment, the first detection module determines the scoliosis risk level of the subject based on the gait video sequence, including: The first detection module inputs the gait video sequence of each angle into the key bone key point recognition model to obtain the bone key point time sequence of the corresponding angle. Each time point in the bone key point time sequence corresponds to all the bone key points contained in a frame of video. The first detection module determines the risk level of scoliosis of the subject based on the time series of key skeletal points from all angles.

[0007] In an optional embodiment, the first detection module determines the scoliosis risk level of the subject based on the time series of skeletal key points from all angles, including: The first detection module determines the 3D skeleton key point sequence based on the time series of skeleton key points from all angles; Gait asymmetry features are calculated based on the 3D skeleton key point sequence. The gait asymmetry features include the trunk coronal plane swing amplitude asymmetry index, pelvic rotation angle asymmetry, stride length and stride frequency asymmetry index, trunk vertical offset change rate, and relative torsion angle between the scapular girdle and pelvic girdle. The risk level of scoliosis in the subject is determined by the gait asymmetry feature.

[0008] In an optional embodiment, the first detection module determines a 3D skeletal keypoint sequence from the time series of skeletal keypoints at all angles, including: For each skeletal keypoint in the skeletal keypoint time series, standard least squares triangulation is used to obtain the initial 3D skeletal keypoint coordinates; The optimized 3D bone key point coordinates are obtained by optimizing the initial 3D bone key point coordinates using the bone length constraint optimization function. The 3D skeleton keypoint sequence is determined based on the optimized 3D skeleton keypoint coordinates.

[0009] In an optional embodiment, determining the subject's scoliosis risk level based on the gait asymmetry features includes: The gait asymmetry features corresponding to the 3D skeletal key point sequence are input into the dynamic scoliosis prediction model to obtain frame-level scoliosis aggregated features and periodic-level scoliosis aggregated features. The global feature vector of scoliosis is obtained by weighting the frame-level aggregated features and the periodic-level aggregated features. The risk level of scoliosis in the subject is determined based on the global feature vector of scoliosis.

[0010] In an optional embodiment, the second detection module generates a scoliosis diagnostic report based on the subject's full spine X-ray, millimeter-wave imaging data, infrared thermal imaging data, and plantar pressure data in a standard static posture, including: The Cobb angle is obtained by X-ray of the entire spine; If the Cobb angle is greater than the preset angle value, the location, curvature classification, and curvature direction of the spinal curvature segment of the subject are determined based on the X-ray of the entire spine. If the Cobb angle is less than or equal to the preset angle value, then the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the pressure center offset are extracted from the millimeter-wave imaging data, infrared thermal imaging data, and plantar pressure data, respectively. The potential risk of scoliosis in the subject is determined based on the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the offset of the pressure center.

[0011] In an optional embodiment, determining whether the subject has a potential risk of scoliosis based on the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the pressure center offset includes: Determine the integral value and direction corresponding to the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the pressure center offset, respectively; The directional score is determined based on the directions corresponding to the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the offset of the pressure center, respectively. The final score is obtained by weighting the integral values ​​and adding the direction scores. The final score is used to determine whether the subject has a potential risk of scoliosis.

[0012] In an optional embodiment, determining whether the subject has a potential risk of scoliosis based on the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the pressure center offset includes: The back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the pressure center offset are combined to form a feature data vector; The feature data vector is input into the static scoliosis prediction model to obtain the scoliosis potential risk prediction result of the subject.

[0013] In an optional embodiment, the second detection module generates a scoliosis diagnostic report based on X-rays of the entire spine, millimeter-wave imaging data, infrared thermal imaging data, and plantar pressure data of the subject in a standard static posture, including: A finite element mesh model of the spine is constructed based on the X-ray images of the entire spine, the millimeter-wave imaging data, the infrared thermal imaging data, and the plantar pressure data. The finite element mesh model of the spine is run to obtain the finite element analysis results, and a scoliosis diagnosis report is generated based on the finite element analysis results.

[0014] In an optional embodiment, constructing a finite element mesh model of the spine based on the X-ray of the entire spine, the millimeter-wave imaging data, the infrared thermal imaging data, and the plantar pressure data includes: A finite element mesh model of the spine was constructed using X-ray images of the entire spine. The millimeter-wave imaging data and the infrared thermal imaging data are respectively registered with the spinal finite element mesh model to obtain the displacement vector of the back surface nodes and the temperature difference of the bilateral paraspinal muscles. Based on the displacement vector of the back surface nodes, the temperature difference of the bilateral paraspinal muscles, the pressure center offset determined by the plantar pressure data, and the pressure asymmetry index of the left and right feet, the displacement boundary conditions, muscle contraction force boundary conditions, and bottom support boundary conditions of the spinal finite element mesh model are set.

[0015] The embodiments of the present invention have the following advantages: This invention provides a three-dimensional spinal posture detection system, comprising: a video acquisition module, a multimodal data acquisition module, a first detection module, a second detection module, and a diagnostic output module. First, the video acquisition module acquires a gait video sequence of the subject in a natural walking state from multiple angles. Then, the first detection module determines the subject's scoliosis risk level based on the gait video sequence. If the scoliosis risk level is greater than a preset level, the multimodal data acquisition module simultaneously acquires X-ray images, millimeter-wave imaging data, infrared thermal imaging data, and plantar pressure data of the subject's entire spine in a standard static posture. Next, the second detection module generates a scoliosis diagnostic report based on the X-ray images, millimeter-wave imaging data, infrared thermal imaging data, and plantar pressure data of the subject's entire spine in a standard static posture. Finally, the diagnostic output module outputs the corresponding scoliosis diagnostic report for the subject. Compared with existing technologies, this application organically integrates dynamic gait analysis with static multimodal data. Specifically, the first detection module preliminarily determines the risk level of scoliosis based on the gait video sequence. If the risk level of scoliosis is greater than the preset level, the second detection module generates a scoliosis diagnosis report based on the acquired static multimodal data. Thus, this application can solve the problem of low accuracy in existing scoliosis detection. Attached Figure Description

[0016] Figure 1This is a flowchart of a three-dimensional posture detection system for detecting scoliosis provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a three-dimensional posture detection system for the spine provided in an embodiment of the present invention. Detailed Implementation

[0017] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0018] Please see Figure 1 This invention provides a three-dimensional spinal posture detection system, comprising: a video acquisition module, a multimodal data acquisition module, a first detection module, a second detection module, and a diagnostic output module. The three-dimensional spinal posture detection system provided in this embodiment can detect scoliosis in a subject. The process of detecting scoliosis using this three-dimensional spinal posture detection system is shown below: S101, through the video acquisition module, acquires gait video sequences of the subject in a natural walking state from multiple angles.

[0019] This embodiment can use a regular smartphone or an RGB camera as the video capture module to collect gait video sequences. Specifically, the subject is required to walk naturally back and forth once on a straight path of approximately 5 meters, with a video duration of approximately 30 seconds. The gait video sequence of the subject in a natural walking state is simultaneously captured from multiple perspectives (front, side, and back).

[0020] S102, the first detection module determines the scoliosis risk level of the subject based on the gait video sequence.

[0021] The scoliosis risk level indicates the risk that an individual is at risk of developing scoliosis. The higher the risk level, the greater the risk of having scoliosis. The risk level can be represented by numbers from 0 to 3, where 0 represents no risk of scoliosis and 3 represents the highest risk of having scoliosis.

[0022] In one optional embodiment provided in this application, the first detection module determines the scoliosis risk level of the subject based on the gait video sequence, including: S1021, the first detection module inputs the gait video sequence of each angle into the key bone key point recognition model to obtain the time sequence of the bone key points of the corresponding angle.

[0023] In this embodiment, each time point in the skeletal keypoint time series corresponds to all the skeletal keypoints contained in one video frame. The key skeletal keypoint recognition model is a pre-trained network classification model. During the training phase, this network classification model uses a large amount of gait data labeled with the gold standard Cobb angle. Through this model, the mapping relationship between gait asymmetry and the severity of scoliosis can be learned. Thus, this model can identify skeletal keypoints in video frames. These skeletal keypoints can be the acromion, anterior superior iliac spine, posterior superior iliac spine, seventh cervical vertebra, sacrum, and other skeletal keypoints closely related to spinal posture. This embodiment does not specifically limit these.

[0024] It should be noted that the Cobb angle is an angle used to measure the severity of scoliosis. The Cobb angle ranges from 0° to 10° and is usually not considered structural scoliosis. When the Cobb angle is greater than 10°, scoliosis can be clearly diagnosed. That is, the larger the Cobb angle, the higher the risk of scoliosis.

[0025] S1022, the first detection module determines the risk level of scoliosis of the subject based on the time series of skeletal key points from all angles.

[0026] Specifically, the first detection module determines the scoliosis risk level of the subject based on the time series of skeletal key points from all angles, including: S10221, the first detection module determines the 3D skeleton key point sequence based on the time series of skeleton key points from all angles.

[0027] More specifically, the first detection module determines the 3D skeletal keypoint sequence based on the time series of skeletal keypoints from all angles, including: obtaining initial 3D skeletal keypoint coordinates for each skeletal keypoint in the time series using standard least squares triangulation; optimizing the initial 3D skeletal keypoint coordinates using a skeletal length constraint optimization function to obtain optimized 3D skeletal keypoint coordinates; and determining the 3D skeletal keypoint sequence based on the optimized 3D skeletal keypoint coordinates.

[0028] In this embodiment, the gait video sequence for each shooting angle corresponds to a skeletal keypoint time sequence. Each angle includes the camera's intrinsic parameters (focal length, principal point, etc.) and extrinsic parameters (rotation, translation, etc.). This embodiment determines the 3D skeletal keypoint sequence based on the skeletal keypoint time sequences of all angles at the same time. This embodiment performs pairwise triangulation on skeletal keypoints of the same type from all viewpoints (e.g., the "left shoulder" skeletal keypoint detected from all viewpoints) to generate a sequence containing all 3D skeletal keypoints at that time.

[0029] In this embodiment, the initial 3D skeleton keypoint coordinates are optimized using a skeleton length constraint optimization function. This involves adjusting the 3D positions of all skeleton keypoints to approximate the initial triangulation result while also satisfying the preset skeleton length constraint. Specifically, the skeleton length constraint optimization function in this embodiment is: Where J represents the total number of key points in the 3D skeleton. Let j be the coordinates of the optimized 3D skeleton keypoint. Let j be the coordinates of the initial 3D skeleton keypoint. The square of the Euclidean distance between the optimized 3D skeleton keypoint coordinates and the initial 3D skeleton keypoint coordinates is used to measure data fidelity; E is the set of human skeleton connection relationships, and each edge e∈E indicates that there is a skeleton connecting two skeleton keypoints. , This represents the numbers of the two skeletal key points connected by edge e. This represents the actual length of the optimized bone e. The standard physiological length of bone e is obtained from population statistics or individual priors. This is a balancing factor (regularization coefficient) used to weigh the importance of data fidelity terms and bone length constraints. This is a bone length constraint term, used to penalize deviations between the optimized bone length and the standard length.

[0030] This embodiment, based on the multi-view triangulation results, utilizes a skeletal length constraint optimization function (i.e., prior human skeletal lengths, such as shoulder width, upper arm length, and forearm length) for secondary optimization. This ensures that the final 3D skeletal keypoint sequence meets reprojection accuracy requirements while also conforming to basic human proportions. Specifically, it minimizes the deviation between the optimized 3D skeletal keypoint coordinates and the initial 3D skeletal keypoint coordinates while penalizing deviations from physiological standard values ​​in skeletal length through a regularization term. This approach forces the skeleton to meet prior human proportions while preserving multi-view geometric observation information. This avoids complex networks and effectively suppresses outliers caused by 2D detection errors in triangulation.

[0031] In one frame, the initial triangulation of the left elbow coordinates had a significant error (due to the back view being obscured by hair), resulting in a calculated "left shoulder - left elbow" bone length of 0.52m, while the standard length is 0.30m. After adding a bone length constraint, the bone length constraint optimization function, while keeping the reprojection error basically unchanged, pulled the left elbow position back to a reasonable range, ultimately achieving a bone length of 0.31m, while the right arm bone length remained almost unchanged.

[0032] S10222, calculate gait asymmetry features based on the 3D skeleton key point sequence.

[0033] The gait asymmetry features include: a trunk coronal plane swing amplitude asymmetry index, which represents the difference in swing amplitude between the left and right acromions relative to the pelvic centerline; pelvic rotation angle asymmetry, which represents the degree of rotational asymmetry between the left and right anterior superior iliac spines in the gait cycle; stride length and gait frequency asymmetry index, which represents the difference in stride length and gait frequency of the right lower limb; trunk vertical offset change rate, which represents the fluctuation characteristics of head vertical displacement in the gait cycle; and relative torsional angle between the scapular girdle and the pelvic girdle, which represents the compensatory rotation of the trunk.

[0034] S10223, determine the risk level of scoliosis of the subject by gait asymmetry characteristics.

[0035] More specifically, determining the scoliosis risk level of the subject through the gait asymmetry features includes: inputting the gait asymmetry features corresponding to the 3D skeletal key point sequence into a dynamic scoliosis prediction model to obtain frame-level scoliosis aggregated features and periodic-level scoliosis aggregated features; performing weighted calculations on the frame-level scoliosis aggregated features and the periodic-level scoliosis aggregated features to obtain a global scoliosis feature vector; and determining the scoliosis risk level of the subject based on the global scoliosis feature vector.

[0036] The frame-level scoliosis aggregation feature is formed by compressing multiple gait asymmetry features within the same frame into a fixed-dimensional vector using a learnable aggregation function (such as an attention mechanism). This vector represents the overall degree of scoliosis abnormality at that moment. The periodic-level scoliosis aggregation feature includes all frames from the heel strike on one side to the heel strike on the same side again. The frame-level aggregation features of all frames within a period are further aggregated (e.g., by averaging, weighting, and / or temporal convolution) to obtain the overall feature vector for that period. This periodic-level scoliosis aggregation feature reflects the overall characteristics of the spinal compensation pattern within that gait period.

[0037] In this embodiment, a global scoliosis feature vector is obtained by weighting the frame-level and periodic-level aggregated scoliosis features. Then, the scoliosis risk level of the subject is determined based on this global feature vector. Because the key skeletal key point recognition model in this embodiment can focus on both instantaneous anomalies in gait (frame-level scoliosis aggregated features) and the overall stable pattern of the period (periodic scoliosis aggregated features), it can more accurately assess scoliosis risk in noisy environments.

[0038] S103 If the risk level of scoliosis is greater than the preset level, the multimodal data acquisition module will simultaneously collect X-ray images, millimeter-wave imaging data, infrared thermal imaging data and plantar pressure data of the whole spine of the subject in a standard static posture.

[0039] The preset level can be set according to actual needs, such as the preset level being 1.

[0040] Specifically, the subject is guided into the data acquisition area and, in a standardized posture (standing naturally, arms hanging naturally, eyes looking forward), static multimodal data is acquired through the multimodal data acquisition module. This multimodal data acquisition module includes an X-ray acquisition module, a millimeter-wave imaging module, an infrared thermal imaging module, and a plantar pressure distribution sensor.

[0041] S104, the second detection module generates a scoliosis diagnosis report based on the X-ray of the whole spine, millimeter-wave imaging data, infrared thermal imaging data and plantar pressure data of the subject in a standard static posture.

[0042] The scoliosis diagnostic report may include: the location of the curved segment of the spine, the type of curvature, the direction of curvature, and the Cobb angle. Specifically, this embodiment can determine the direction of curvature by combining infrared thermal imaging data and plantar pressure data. If the center of pressure shifts to the right (shift ≥ 5 mm), it indicates that the curve bulges to the right; if the center of pressure shifts to the left (shift ≥ 5 mm), it indicates that the curve bulges to the left. If the temperature of the right back region is higher than that of the left back region (temperature difference ≥ 0.5°C), it indicates that the curve bulges to the right; if the temperature of the left back region is higher than that of the right back region (temperature difference ≥ 0.5°C), it indicates that the curve bulges to the left.

[0043] In one optional embodiment provided in this application, the second detection module generates a scoliosis diagnostic report based on the X-ray of the entire spine, millimeter-wave imaging data, infrared thermal imaging data, and plantar pressure data of the subject in a standard static posture, including: S1041, the Cobb angle is obtained through X-ray of the entire spine.

[0044] S1042, if the Cobb angle is greater than the preset angle value, then the position, curvature classification, and curvature direction of the spinal curvature segment of the subject are determined based on the X-ray of the entire spine.

[0045] The preset angle value is set according to actual needs; in this embodiment, the preset angle value can be 10°. Specifically, in this embodiment, several types of vertebrae are located on X-rays of the entire spine, such as apical vertebrae, end vertebrae, neutral vertebrae, and stable vertebrae. Then, based on these types of vertebrae, the position, curvature classification, curvature direction, and Cobb angle of the spinal curvature segment are determined. For example, the curvature position is determined by the anatomical segment where the apical vertebra is located, such as the thoracic curve (T2-T11), thoracolumbar curve (T12-L1), and lumbar curve (L2-L4). The curve with the largest Cobb angle is selected as the structural principal curve.

[0046] S1043, if the Cobb angle is less than or equal to the preset angle value, then extract the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the pressure center offset from the millimeter-wave imaging data, infrared thermal imaging data, and plantar pressure data, respectively.

[0047] In this embodiment, if the Cobb angle is determined to be less than or equal to a preset angle value, it is not possible to directly determine whether scoliosis has occurred based solely on the Cobb angle. However, the abnormal synergy of millimeter-wave captured back asymmetric rotation, infrared thermography reflecting compensatory temperature differences in the paraspinal muscles, and plantar pressure indicating a shift in the center of gravity, is often an early sign of spinal biomechanical instability and a crucial intervention window for adolescents progressing from poor posture to structural scoliosis. Therefore, after determining that the Cobb angle is less than or equal to the preset angle value, this embodiment also needs to determine whether the subject has a potential risk of scoliosis based on the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the shift in the center of pressure. This potential risk of scoliosis indicates whether the subject has a potential risk of scoliosis or is already at risk of having scoliosis.

[0048] In this embodiment, a three-dimensional curved surface model of the back reconstructed by millimeter-wave imaging is used to automatically calculate the torso rotation angle, and the maximum value of the entire back is taken as the back asymmetry angle (ATR). A thermal image of the back is acquired by an infrared thermal imager, and the back is divided into two regions, left and right, along the midline of the spine. The difference in the average temperature of all pixels in the two regions is calculated, and the absolute value is taken as the maximum temperature difference (ΔT) between the two sides of the spine. The plantar pressure distribution matrix of the subject when standing naturally is acquired by a plantar pressure sensor, and the distance between the coordinates of the pressure center and the geometric center of the foot is calculated to obtain the pressure center offset (COP offset).

[0049] S1044, determine whether the subject has a potential risk of scoliosis based on the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the pressure center offset.

[0050] Optionally, determining whether the subject has a potential risk of scoliosis based on the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the pressure center offset includes: determining the integral value and direction corresponding to the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the pressure center offset, respectively; determining the direction score based on the direction corresponding to the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the pressure center offset, respectively; performing a weighted calculation on the integral value and adding the direction score to obtain the final score; and determining whether the subject has a potential risk of scoliosis based on the final score.

[0051] Specifically, in this embodiment, the integral values ​​corresponding to the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the pressure center offset can be obtained through a preset mapping table. This preset mapping table is shown in Table 1 below: Table 1 In this embodiment, when the directions corresponding to the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the pressure center offset are consistent (i.e., the maximum ATR area, the side with higher ΔT temperature, and the COP offset direction all point to the same side), it indicates a higher possibility of structural potential scoliosis, and 1 point is added to the total score as a direction score.

[0052] For example, ATR=8° (right T7-T11 region), ΔT=0.9°C (right side higher than left side), COP offset=9mm (offset to the right). Matching using the preset mapping table yields the following results: ATR=8° corresponds to moderate abnormality (2 points), weighted score=2×2=4 points; ΔT=0.9°C corresponds to moderate abnormality (2 points), weighted score=1.5×2=3 points; COP=9mm corresponds to moderate abnormality (2 points), weighted score=1×2=2 points; the directions of the right side of TR, the right side of ΔT, and the right side of COP are completely consistent, i.e., the direction score is 1 point, and the final score=4+3+2+1=10 points. Since the final score is ≥8 points, a potential risk of right thoracic scoliosis can be diagnosed.

[0053] Optionally, determining whether the subject has a potential risk of scoliosis based on the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the pressure center offset includes: constructing a feature data vector from the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the pressure center offset; and inputting the feature data vector into a static scoliosis prediction model to obtain a prediction result of the subject's potential scoliosis risk. The static scoliosis prediction model is a pre-trained classification model, through which the prediction result of the subject's potential scoliosis risk can be obtained.

[0054] In another optional embodiment provided in this application, the second detection module generates a scoliosis diagnostic report based on X-rays of the entire spine, millimeter-wave imaging data, infrared thermal imaging data, and plantar pressure data of the subject in a standard static posture, including: S104A, a finite element mesh model of the spine is constructed based on the X-ray of the entire spine, the millimeter-wave imaging data, the infrared thermal imaging data, and the plantar pressure data.

[0055] Specifically, the step of constructing a finite element mesh model of the spine based on the X-ray images of the entire spine, the millimeter-wave imaging data, the infrared thermal imaging data, and the plantar pressure data includes: S104A1, a finite element mesh model of the spine is constructed using X-ray images of the entire spine.

[0056] Specifically, a semi-automatic segmentation algorithm can be used to extract the contour of each vertebra from anteroposterior and lateral X-ray images. By matching the anteroposterior and lateral contours, a 3D point cloud of each vertebra is reconstructed. A statistical shape model is used to fill in the missing dorsal information, generating a complete 3D vertebral surface. Intervertebral disc geometry is created between adjacent vertebrae. The height is taken as the distance between the superior and inferior endplates, and the cross-sectional area is the average area of ​​the superior and inferior endplates. The intervertebral disc is divided into the nucleus pulposus and the annulus fibrosus. Based on anatomical location, anterior longitudinal ligaments, posterior longitudinal ligaments, ligamentum flavum, interspinous ligaments, and supraspinous ligaments are added between the vertebrae, represented by cable elements or spring elements, thus obtaining a finite element mesh model of the spine.

[0057] S104A2, the millimeter-wave imaging data and the infrared thermal imaging data are registered with the finite element mesh model of the spine to obtain the displacement vector of the back surface nodes and the temperature difference of the bilateral paraspinal muscles.

[0058] Among them, the displacement vector of the back surface nodes is used to quantify the vector of positional change of specific points (nodes) on the back surface in three-dimensional space. In the finite element mesh model analysis of the spine, it represents the forced deformation of the back surface of the model to conform to the real surface morphology scanned by equipment such as millimeter-wave radar. The bilateral paraspinal muscle temperature difference is the temperature difference calculated after measuring the surface temperature of the paraspinal muscles (such as trapezius and latissimus dorsi) on both sides of the spine using infrared thermal imaging technology.

[0059] S104A3, based on the displacement vector of the back surface nodes, the temperature difference of the bilateral paraspinal muscles, and the pressure center offset determined by the plantar pressure data, and the pressure asymmetry index of the left and right feet, the displacement boundary conditions, muscle contraction force boundary conditions, and bottom support boundary conditions of the finite element mesh model of the spine are set.

[0060] In this embodiment, a triangular mesh of the back surface is generated based on millimeter-wave imaging data, serving as the outer surface of the spinal finite element mesh model. The surface mesh is aligned to the spinal finite element mesh model using rigid registration (based on common landmarks such as the T1 spinous process and sacrum). After registration, the spinal finite element mesh model is located inside the surface mesh. Nodes on the surface mesh corresponding to the spinal projection area are selected, and their displacement degrees of freedom are forcibly set to the actual positions measured by the millimeter-wave imaging. After registration of the millimeter-wave imaging data with the standard template, each surface node i has a three-dimensional displacement vector. In the back surface mesh of the spinal finite element mesh model, a set of nodes corresponding to the millimeter-wave measurement points is selected, and the displacement degrees of freedom of these nodes are directly fixed to the measured values. This forces the back surface of the model to produce deformation consistent with that of a real human body, thereby driving the spine to produce corresponding lateral curvature.

[0061] The process of setting the boundary conditions for muscle contraction force in the finite element mesh model of the spine based on the temperature difference of the bilateral paraspinal muscles is as follows: the temperature difference ΔT between the bilateral paraspinal muscles is transformed into an asymmetric distribution of muscle contraction force on both sides. Opposite forces are applied to the equivalent muscle elements on the convex and concave sides. The calculation formula is as follows: Where ΔT is the temperature difference of the bilateral paraspinal muscles measured by infrared thermal imaging, and k is an empirical proportionality coefficient used to convert the temperature difference into force imbalance. , These are the muscle force distribution coefficients for the convex and concave sides, respectively; Total muscle strength estimated based on body weight; , These are the nodal forces applied to the convex side (outward pulling) and the concave side (inward pulling or a smaller force), respectively.

[0062] In this embodiment, the bottom support boundary conditions are divided into two parts: bending moment boundary (determined based on the offset of the pressure center) and asymmetric vertical support boundary (determined based on the pressure asymmetry index of the left and right feet).

[0063] It should be noted that when a person stands, the shift in the center of pressure means that the projection of the center of gravity deviates from the midline of the spine. This generates a lateral bending moment at the base of the spine, causing the entire spine to tilt in the direction of the shift. Therefore, in this embodiment, the process of setting the bending moment boundary of the finite element mesh model of the spine is as follows: a rigid reference point is created at the base of the sacrum or below the pelvis in the finite element mesh model of the spine. The nodes on the surface of the base of the sacrum are constrained to this reference point through motion coupling or distributed coupling, so that the bottom nodes have no relative motion with respect to the reference point. A bending moment M is then directly applied to the reference point. The direction is consistent with the direction of the pressure center offset (for example, if the pressure center offset is positive to the right, then the bending moment rotates clockwise around the front and rear axes, causing the spine to tilt to the right). Among these, denoted as the pressure center offset (i.e., the horizontal offset distance of the pressure center relative to the geometric center of the foot, with direction (negative on the left, positive on the right)), and W is the weight of the subject being tested.

[0064] The asymmetry of pressure between the left and right feet leads to uneven distribution of ground reaction forces on both sides, causing pelvic tilt and transmitting asymmetric loads upwards. The process of setting the asymmetric vertical support boundary of the spinal finite element mesh model is as follows: The nodes at the bottom of the sacrum or pelvis are divided into left and right regions. A vertically upward distributed force is applied to the left region, making the resultant force equal to the vertical support force on the left side. Similarly, a vertically upward distributed force is applied to the right region, making the resultant force equal to the vertical support force on the right side. Specifically, the vertical support force on the left side is equal to the proportion of the pressure on the left foot to the total pressure multiplied by the subject's weight, and the vertical support force on the right side is equal to the proportion of the pressure on the right foot to the total pressure multiplied by the subject's weight.

[0065] S104B, Run the finite element mesh model of the spine to obtain the finite element analysis results, and generate a scoliosis diagnosis report based on the finite element analysis results.

[0066] S105, the diagnostic output module outputs the scoliosis diagnostic report corresponding to the tested person.

[0067] This embodiment provides a three-dimensional spinal posture detection system, which includes a video acquisition module, a multimodal data acquisition module, a first detection module, a second detection module, and a diagnostic output module. First, the video acquisition module acquires a gait video sequence of the subject in a natural walking state from multiple angles. Then, the first detection module determines the subject's scoliosis risk level based on the gait video sequence. If the scoliosis risk level is greater than a preset level, the multimodal data acquisition module simultaneously acquires X-ray images, millimeter-wave imaging data, infrared thermal imaging data, and plantar pressure data of the subject's entire spine in a standard static posture. Next, the second detection module generates a scoliosis diagnostic report based on the X-ray images, millimeter-wave imaging data, infrared thermal imaging data, and plantar pressure data of the subject's entire spine in a standard static posture. Finally, the diagnostic output module outputs the corresponding scoliosis diagnostic report for the subject. Compared with existing technologies, this application organically integrates dynamic gait analysis with static multimodal data. Specifically, the first detection module preliminarily determines the risk level of scoliosis based on the gait video sequence. If the risk level of scoliosis is greater than the preset level, the second detection module generates a scoliosis diagnosis report based on the acquired static multimodal data. Thus, this application can solve the problem of low accuracy in existing scoliosis detection.

[0068] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0069] In one embodiment, a three-dimensional posture detection system for the spine is provided. Figure 2 As shown, the system includes: a video acquisition module 21, a multimodal data acquisition module 22, a first detection module 23, a second detection module 24, and a diagnostic output module 25; Video acquisition module 21 is used to acquire gait video sequences of the subject in a natural walking state from multiple angles; The first detection module 23 is used to determine the scoliosis risk level of the subject based on the gait video sequence; The multimodal data acquisition module 22 is used to simultaneously acquire X-ray images, millimeter-wave imaging data, infrared thermal imaging data, and plantar pressure data of the subject's entire spine in a standard static posture if the risk level of scoliosis is greater than the preset level. The second detection module 24 is used to generate a scoliosis diagnosis report based on the X-ray of the whole spine of the subject in a standard static posture, millimeter-wave imaging data, infrared thermal imaging data and plantar pressure data. The diagnostic output module 25 is used to output a scoliosis diagnostic report corresponding to the subject being tested.

[0070] In an optional embodiment, the first detection module 23 is specifically used for: The first detection module inputs the gait video sequence of each angle into the key bone key point recognition model to obtain the bone key point time sequence of the corresponding angle. Each time point in the bone key point time sequence corresponds to all the bone key points contained in a frame of video. The first detection module determines the risk level of scoliosis of the subject based on the time series of key skeletal points from all angles.

[0071] In an optional embodiment, the first detection module 23 is specifically used for: The first detection module determines the 3D skeleton key point sequence based on the time series of skeleton key points from all angles; Gait asymmetry features are calculated based on the 3D skeleton key point sequence. The gait asymmetry features include the trunk coronal plane swing amplitude asymmetry index, pelvic rotation angle asymmetry, stride length and stride frequency asymmetry index, trunk vertical offset change rate, and relative torsion angle between the scapular girdle and pelvic girdle. The risk level of scoliosis in the subject is determined by the gait asymmetry feature.

[0072] In an optional embodiment, the first detection module 23 is specifically used for: For each skeletal keypoint in the skeletal keypoint time series, standard least squares triangulation is used to obtain the initial 3D skeletal keypoint coordinates; The optimized 3D bone key point coordinates are obtained by optimizing the initial 3D bone key point coordinates using the bone length constraint optimization function. The 3D skeleton keypoint sequence is determined based on the optimized 3D skeleton keypoint coordinates.

[0073] In an optional embodiment, the first detection module 23 is specifically used for: The gait asymmetry features corresponding to the 3D skeletal key point sequence are input into the dynamic scoliosis prediction model to obtain frame-level scoliosis aggregated features and periodic-level scoliosis aggregated features. The global feature vector of scoliosis is obtained by weighting the frame-level aggregated features and the periodic-level aggregated features. The risk level of scoliosis in the subject is determined based on the global feature vector of scoliosis.

[0074] In an optional embodiment, the second detection module 24 is specifically used for: The Cobb angle is obtained by X-ray of the entire spine; If the Cobb angle is greater than the preset angle value, the location, curvature classification, and curvature direction of the spinal curvature segment of the subject are determined based on the X-ray of the entire spine. If the Cobb angle is less than or equal to the preset angle value, then the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the pressure center offset are extracted from the millimeter-wave imaging data, infrared thermal imaging data, and plantar pressure data, respectively. The potential risk of scoliosis in the subject is determined based on the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the offset of the pressure center.

[0075] In an optional embodiment, the second detection module 24 is specifically used for: Determine the integral value and direction corresponding to the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the pressure center offset, respectively; The directional score is determined based on the directions corresponding to the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the offset of the pressure center, respectively. The final score is obtained by weighting the integral values ​​and adding the direction scores. The final score is used to determine whether the subject has a potential risk of scoliosis.

[0076] In an optional embodiment, the second detection module 24 is specifically used for: The back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the pressure center offset are combined to form a feature data vector; The feature data vector is input into the static scoliosis prediction model to obtain the scoliosis potential risk prediction result of the subject.

[0077] In an optional embodiment, the second detection module 24 is specifically used for: A finite element mesh model of the spine is constructed based on the X-ray images of the entire spine, the millimeter-wave imaging data, the infrared thermal imaging data, and the plantar pressure data. The finite element mesh model of the spine is run to obtain the finite element analysis results, and a scoliosis diagnosis report is generated based on the finite element analysis results.

[0078] In an optional embodiment, the second detection module 24 is specifically used for: A finite element mesh model of the spine was constructed using X-ray images of the entire spine. The millimeter-wave imaging data and the infrared thermal imaging data are respectively registered with the spinal finite element mesh model to obtain the displacement vector of the back surface nodes and the temperature difference of the bilateral paraspinal muscles. Based on the displacement vector of the back surface nodes, the temperature difference of the bilateral paraspinal muscles, the pressure center offset determined by the plantar pressure data, and the pressure asymmetry index of the left and right feet, the displacement boundary conditions, muscle contraction force boundary conditions, and bottom support boundary conditions of the spinal finite element mesh model are set.

[0079] It should be noted that the above detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0080] Each module in the aforementioned device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0081] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.

[0082] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A three-dimensional posture detection system for the spine, characterized in that, The system includes: a video acquisition module, a multimodal data acquisition module, a first detection module, a second detection module, and a diagnostic output module; The video acquisition module acquires gait video sequences of the subject in a natural walking state from multiple angles. The first detection module determines the scoliosis risk level of the subject based on the gait video sequence; If the risk level of scoliosis is greater than the preset level, the multimodal data acquisition module will simultaneously acquire X-ray images, millimeter-wave imaging data, infrared thermal imaging data, and plantar pressure data of the subject's entire spine in a standard static posture. The second detection module generates a scoliosis diagnosis report based on the X-ray of the entire spine, millimeter-wave imaging data, infrared thermal imaging data, and plantar pressure data of the subject in a standard static posture. The diagnostic output module outputs a scoliosis diagnostic report for the person being tested.

2. The system according to claim 1, characterized in that, The first detection module determines the scoliosis risk level of the subject based on the gait video sequence, including: The first detection module inputs the gait video sequence of each angle into the key bone key point recognition model to obtain the bone key point time sequence of the corresponding angle. Each time point in the bone key point time sequence corresponds to all the bone key points contained in a frame of video. The first detection module determines the risk level of scoliosis of the subject based on the time series of key skeletal points from all angles.

3. The system according to claim 2, characterized in that, The first detection module determines the scoliosis risk level of the subject based on the time series of skeletal key points from all angles, including: The first detection module determines the 3D skeleton key point sequence based on the time series of skeleton key points from all angles; Gait asymmetry features are calculated based on the 3D skeleton key point sequence. The gait asymmetry features include the trunk coronal plane swing amplitude asymmetry index, pelvic rotation angle asymmetry, stride length and stride frequency asymmetry index, trunk vertical offset change rate, and relative torsion angle between the scapular girdle and pelvic girdle. The risk level of scoliosis in the subject is determined by the gait asymmetry feature.

4. The system according to claim 3, characterized in that, The first detection module determines the 3D skeletal keypoint sequence based on the time series of skeletal keypoints from all angles, including: For each skeletal keypoint in the skeletal keypoint time series, standard least squares triangulation is used to obtain the initial 3D skeletal keypoint coordinates; The optimized 3D bone key point coordinates are obtained by optimizing the initial 3D bone key point coordinates using the bone length constraint optimization function. The 3D skeleton keypoint sequence is determined based on the optimized 3D skeleton keypoint coordinates.

5. The system according to claim 3, characterized in that, The determination of the subject's scoliosis risk level based on the gait asymmetry features includes: The gait asymmetry features corresponding to the 3D skeletal key point sequence are input into the dynamic scoliosis prediction model to obtain frame-level scoliosis aggregated features and periodic-level scoliosis aggregated features. The global feature vector of scoliosis is obtained by weighting the frame-level aggregated features and the periodic-level aggregated features. The risk level of scoliosis in the subject is determined based on the global feature vector of scoliosis.

6. The system according to any one of claims 1-5, characterized in that, The second detection module generates a scoliosis diagnosis report based on the X-ray, millimeter-wave imaging data, infrared thermal imaging data, and plantar pressure data of the subject's entire spine in a standard static posture, including: The Cobb angle is obtained by X-ray of the entire spine; If the Cobb angle is greater than the preset angle value, the location, curvature classification, and curvature direction of the spinal curvature segment of the subject are determined based on the X-ray of the entire spine. If the Cobb angle is less than or equal to the preset angle value, then the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the pressure center offset are extracted from the millimeter-wave imaging data, infrared thermal imaging data, and plantar pressure data, respectively. The potential risk of scoliosis in the subject is determined based on the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the offset of the pressure center.

7. The system according to claim 6, characterized in that, The determination of whether the subject has a potential risk of scoliosis based on the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the pressure center offset includes: Determine the integral value and direction corresponding to the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the pressure center offset, respectively; The directional score is determined based on the directions corresponding to the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the offset of the pressure center, respectively. The final score is obtained by weighting the integral values ​​and adding the direction scores. The final score is used to determine whether the subject has a potential risk of scoliosis.

8. The system according to claim 6, characterized in that, The determination of whether the subject has a potential risk of scoliosis based on the back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the pressure center offset includes: The back asymmetry angle, the maximum temperature difference between the two sides of the spine, and the pressure center offset are combined to form a feature data vector; The feature data vector is input into the static scoliosis prediction model to obtain the scoliosis potential risk prediction result of the subject.

9. The system according to any one of claims 1-5, characterized in that, The second detection module generates a scoliosis diagnosis report based on the X-ray, millimeter-wave imaging data, infrared thermal imaging data, and plantar pressure data of the subject's entire spine in a standard static posture, including: A finite element mesh model of the spine is constructed based on the X-ray images of the entire spine, the millimeter-wave imaging data, the infrared thermal imaging data, and the plantar pressure data. The finite element mesh model of the spine is run to obtain the finite element analysis results, and a scoliosis diagnosis report is generated based on the finite element analysis results.

10. The system according to claim 9, characterized in that, The construction of a finite element mesh model of the spine based on the X-ray images of the entire spine, the millimeter-wave imaging data, the infrared thermal imaging data, and the plantar pressure data includes: A finite element mesh model of the spine was constructed using X-ray images of the entire spine. The millimeter-wave imaging data and the infrared thermal imaging data are respectively registered with the spinal finite element mesh model to obtain the displacement vector of the back surface nodes and the temperature difference of the bilateral paraspinal muscles. Based on the displacement vector of the back surface nodes, the temperature difference of the bilateral paraspinal muscles, the pressure center offset determined by the plantar pressure data, and the pressure asymmetry index of the left and right feet, the displacement boundary conditions, muscle contraction force boundary conditions, and bottom support boundary conditions of the spinal finite element mesh model are set.

Images