Method and system for scoliosis detection based on depth camera and infrared thermal imaging

Abstract

The present application relates to the technical field of medical image processing and artificial intelligence assisted diagnosis, especially to a scoliosis detection method and system based on a depth camera and infrared thermal imaging, the method comprising: synchronously collecting depth images and infrared thermal imaging data of the back of a subject; generating a three-dimensional point cloud from the depth images and extracting back surface topographic features; extracting bilateral temperature distribution features from the infrared images and calculating temperature asymmetry index and bilateral temperature difference entropy; using a multi-head bidirectional cross-attention mechanism to fuse the depth morphological features and the infrared thermal features into a fusion feature vector; inputting the fusion feature vector into a multi-task classification regression network based on a Transformer encoder, simultaneously outputting the scoliosis type and Cobb angle estimate value, and generating a detection report; the present application realizes adaptive fusion of depth morphological features and infrared thermal features, has high sensitivity for mild scoliosis detection, and is suitable for large-scale scoliosis rapid screening without ionizing radiation.

Description

Technical Field

[0001] This invention relates to the field of medical image processing and artificial intelligence-assisted diagnosis technology, specifically to a method and system for detecting scoliosis based on depth cameras and infrared thermal imaging. Background Technology

[0002] Scoliosis is a three-dimensional deformity of the spine, characterized by lateral curvature in the coronal plane and vertebral rotation in the sagittal plane. It is one of the most common spinal deformities in adolescents. If not detected and intervened in time, scoliosis can lead to abnormal posture, chronic pain, impaired cardiopulmonary function, and even neurological dysfunction.

[0003] Chinese invention patent application publication number CN121059145A discloses a method for detecting adolescent scoliosis based on three-dimensional imaging. This method acquires multimodal data such as back depth map, temperature distribution, posture angle, and plantar pressure matrix of the subject through a depth camera, infrared imaging device, IMU sensor, and plantar pressure distribution sensor. A cloud platform generates a three-dimensional point cloud and uses a Poisson surface reconstruction algorithm to obtain a three-dimensional curved surface model of the back. The overall posture is corrected by combining posture angle and pressure center offset, and a thermal correction model of local deformation is constructed based on a heat map. The method automatically identifies spinal segment landmarks, fits the spinal midline, calculates the Cobb angle, determines the degree of scoliosis, and outputs an analysis report through a deep learning algorithm. This invention does not require X-ray imaging, has high detection safety, can integrate posture, mechanical and thermal information, reduces human posture interference, and is suitable for early screening and dynamic re-examination of adolescents.

[0004] In the aforementioned methods, infrared thermal imaging data is merely used as a unidirectional correction term for the three-dimensional geometric model, failing to achieve deep fusion at the feature level. It projects and interpolates thermal map temperature data onto the vertices of the three-dimensional curved surface model of the back, defining temperature-driven normal deformation in abnormal areas. Essentially, this is a unidirectional mapping from "temperature to geometry," with infrared data passively serving as a local correction tool for the geometric model. This approach cannot adaptively adjust the contribution weights of different modalities based on the characteristics of the input data. In mild scoliosis scenarios where surface morphology differences are not significant, the information on paraspinal muscle metabolic asymmetry contained in infrared thermal imaging is difficult to utilize effectively. Furthermore, this approach only uses a local temperature difference exceeding 1.5℃ as the anomaly threshold, failing to establish a systematic quantitative index system for bilateral temperature asymmetry, resulting in a single dimension of temperature data analysis. Simultaneously, its detection task only outputs the Cobb angle, failing to provide scoliosis type classification results, making it difficult to meet the clinical need for comprehensive diagnostic information. Moreover, this approach requires wearing an IMU sensor and using a plantar pressure pad, making the operation complex and unsuitable for large-scale rapid screening scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for detecting scoliosis based on depth cameras and infrared thermal imaging. It introduces a multi-head bidirectional cross-attention mechanism to achieve adaptive fusion of depth morphological features and infrared thermal features, which has high sensitivity for detecting mild scoliosis and reduces the overall rate of missed diagnosis and misdiagnosis.

[0006] To achieve the above objectives, the present invention provides the following technical solution: In a first aspect, the present invention provides a method for detecting scoliosis based on a depth camera and infrared thermal imaging, comprising the following steps: Step S1: Acquire depth image data and infrared thermal imaging data of the subject's back; the depth image data is acquired by a depth camera at a preset shooting distance, and the infrared thermal imaging data is acquired simultaneously by an infrared thermal imager. Step S2: Preprocess the depth image data to generate a three-dimensional point cloud of the back, and extract the surface morphology features of the back based on the three-dimensional point cloud of the back. Step S3: Preprocess the infrared thermal imaging data, extract the temperature distribution characteristics on both sides of the back, and calculate the temperature asymmetry index on both sides. Step S4: Perform multimodal feature fusion of the back surface morphology features and the bilateral temperature asymmetry index to construct a fused feature vector; Step S5: Input the fused feature vector into the preset scoliosis classification and detection model, and output the scoliosis detection result; Step S6: Based on the scoliosis detection results, generate a detection report that includes the type of scoliosis, the assessment level of the degree of scoliosis, and recommended measures.

[0007] Preferably, in step S1, synchronous acquisition specifically involves: pre-calibrating the spatial transformation matrix between the depth camera and the infrared thermal imager, and during acquisition, controlling the synchronous exposure of the depth camera and the infrared thermal imager by a synchronous trigger signal.

[0008] Preferably, in step S2, the depth image data is preprocessed to generate a back 3D point cloud, specifically including: Denoising processing is performed on the depth image data, and a bilateral filtering algorithm is used to remove artifacts and noise points in the depth image; The denoised depth image is converted into a 3D point cloud using the intrinsic parameter matrix of the depth camera; A RANSAC-based planar segmentation algorithm is used to segment and remove the background plane from the 3D point cloud, while retaining the point cloud of the back region. Voxel downsampling was performed on the point cloud of the back region to obtain a uniform density 3D point cloud of the back.

[0009] Preferably, in step S2, extracting the surface topography features of the back based on the three-dimensional point cloud of the back specifically includes: The curvature features of the back surface, including Gaussian curvature and mean curvature, are calculated from three-dimensional points on the back. The midsagittal line of the back was extracted using a symmetry-based analysis method, and the back was divided into left and right regions based on the midsagittal line. Calculate the surface morphology difference index between the left and right regions, including: the mean surface height difference between the left and right sides. The maximum height difference between the left and right sides And the asymmetry ratio of the projected areas on the left and right sides. ; The relative height and rotation angle features of the scapular region are extracted. The scapular region is located by the curvature extremum point. The vertical height difference between the left and right scapulae in the YZ coronal plane and the rotation angle difference relative to the midline sagittal line are calculated.

[0010] Preferably, in step S3, extracting the temperature distribution features on both sides of the back specifically includes: Based on the pre-calibrated spatial transformation matrix between the depth camera and the infrared thermal imager, the acquired two-dimensional temperature data is mapped to the three-dimensional point cloud space on the back, so that each three-dimensional surface point on the back has a corresponding temperature attribute value. On the registered back 3D point cloud, the left and right temperature regions are divided with the median sagittal line as the reference. Calculate the statistical temperature characteristics of the left and right temperature regions respectively, including: the average temperature of the left side. and the average temperature on the right Standard deviation of temperature on the left side and the standard deviation of temperature on the right Maximum temperature on the left side and the maximum temperature on the right Minimum temperature on the left and the minimum temperature on the right ; The back is divided into multiple transverse bands along the midsagittal line, and the temperature gradient values ​​on the left and right sides of each band are calculated.

[0011] Preferably, in step S3, calculating the bilateral temperature asymmetry index specifically involves: Constructing a two-sided temperature asymmetry index The calculation formula is: ; in, The number of strip-shaped regions, For the first Average temperature on the left side of the strip-shaped region; For the first Average temperature on the right side of the strip-shaped region; The value range is [0, 1]; Simultaneously calculate the entropy of the temperature difference between the two sides. The calculation formula is: ; in, For the first The absolute value of the temperature difference between corresponding points on the left and right sides of each pixel; This represents the normalized probability density of the temperature difference distribution; To prevent small constants from taking the value of 0 in logarithmic operations; This represents the number of intervals in the temperature difference histogram.

[0012] Preferably, in step S4, the multimodal feature fusion specifically includes: Constructing the first feature vector Includes: average surface height difference Maximum height difference Area asymmetry ratio Relative height of the left scapula Relative height of the right scapula Difference in rotation angle of the scapula relative to the midline ; Constructing the second feature vector Includes: bilateral temperature asymmetry index Two-sided temperature difference entropy Local temperature difference in each zone; right and Perform Z-score normalization to make the mean 0 and the standard deviation 1 respectively; Employing a multi-head attention mechanism and Perform cross-attention calculation: As a query vector Attention weights are calculated using the key vector and value vector; simultaneously... As a query vector The inverse attention weights are calculated using the key and value vectors, and the bidirectional attention output is residually connected to the original features and normalized. Finally, the fused feature vector is obtained by projection through a fully connected layer. .

[0013] Preferably, in step S5, the scoliosis classification and detection model is a multi-task classification and regression network based on the Transformer encoder architecture, the structure of which includes: Input layer: Receives the fused feature vector ; Location encoding layer: Adds learnable location encoding to the feature dimension; 2-4 Transformer encoder layers, each containing a multi-head self-attention sublayer and a feedforward neural network sublayer; Classification Header: Outputs the probability distribution of four types of scoliosis: normal, C-type thoracic, C-type lumbar, and S-type. Regression head: Outputs the estimated Cobb angle.

[0014] Preferably, the model is trained using classification loss. and regression loss The weighted sum is used as the total loss function, and the calculation formula is as follows: ; in, and These are the weighting coefficients, and The classification loss uses the cross-entropy loss function, and the regression loss uses the smoothing L1 loss function.

[0015] In a second aspect, the present invention provides a scoliosis detection system based on a depth camera and infrared thermal imaging, comprising: The data acquisition module, including a depth camera and an infrared thermal imager, is used to simultaneously acquire depth image data and infrared thermal imaging data of the subject's back. The depth data processing module is used to preprocess the depth image data to generate a three-dimensional point cloud of the back and extract the surface morphology features of the back. The infrared data processing module is used to preprocess the infrared thermal imaging data, extract the temperature distribution characteristics on both sides of the back, and calculate the temperature asymmetry index on both sides. A multimodal fusion module is used to fuse the back surface topography features with the bilateral temperature asymmetry index to construct a fused feature vector. The classification and detection module has a built-in trained scoliosis classification and detection model, which is used to output scoliosis detection results based on the fused feature vector. The report generation module is used to generate a test report based on the scoliosis test results, which includes the type of scoliosis, the assessment level of the degree of scoliosis, and recommended measures.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention introduces a multi-head bidirectional cross-attention mechanism into scoliosis detection to... As a query, Attention weights are calculated using both Key and Value; simultaneously... As a query, The reverse attention weights are calculated as Key and Value. The bidirectional attention mechanism enables equal and bidirectional information interaction between the two modalities at the feature level. The model can adaptively adjust the contribution weights of different modalities according to the characteristics of the input data: when the surface morphology difference is significant, the weight of depth morphology features is automatically increased; when the surface morphology difference is not obvious (such as slight lateral bending), the weight of infrared thermal features is automatically increased.

[0017] 2. This invention proposes a temperature asymmetry index. and the entropy of the two-sided temperature difference Two quantitative indicators, The overall degree of bilateral temperature asymmetry is quantitatively characterized by normalized weighted summation of bilateral temperature differences in multiple band-shaped regions along the spinal direction. The probability density entropy value of temperature difference distribution can be used to quantitatively characterize the degree of unevenness in temperature distribution. Using both methods together can comprehensively quantify the degree of asymmetry in paraspinal muscle metabolism and its correlation with the severity of scoliosis, thus aiding in the early detection of scoliosis.

[0018] 3. This invention uses a Transformer encoder as a shared backbone network and sets up a classification head and a regression head in parallel. The classification head outputs the probability distribution of four types of scoliosis: normal, C-type thoracic, C-type lumbar, and S-type. The regression head outputs the estimated Cobb angle. Both share the Transformer encoder and are jointly trained by weighted summation of classification loss and regression loss, so that classification and regression results can be obtained simultaneously.

[0019] 4. This invention uses two non-contact optical sensors: a depth camera and an infrared thermal imager. It does not require any devices to be worn on the subject's body surface or to be used in conjunction with foot pressure sensors. A single test only takes a few seconds to tens of seconds, simplifying the testing process and making it suitable for large-scale rapid screening scenarios such as school screening and community health checkups.

[0020] 5. This invention automates the entire process from data acquisition, posture guidance, feature extraction, multimodal fusion to result output, eliminating the need for manual readings by physicians, thus eliminating subjective judgment differences and ensuring good reproducibility of test results. Simultaneously, the system incorporates a posture guidance module and a data quality control module to ensure the standardization and consistency of the acquired data. Attached Figure Description

[0021] Figure 1 This is a flowchart of the method in this invention; Figure 2 This is a schematic diagram of the device for synchronous acquisition by a depth camera and an infrared thermal imager in this invention. Figure 3 This is a flowchart of the depth image preprocessing and 3D point cloud generation in this invention; Figure 4This is a schematic diagram of the extraction of back surface morphology features in this invention; Figure 5 This is a schematic diagram of the spatial registration of infrared thermal imaging data and the division of bilateral temperature regions in this invention. Figure 6 This is a diagram of the multimodal feature fusion network structure in this invention; Figure 7 This is a structural diagram of the scoliosis classification and detection model in this invention; Figure 8 This is an example interface diagram of the test report in this invention; Figure 9 This is a system structure block diagram of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Example 1, Detection method.

[0024] Please see Figures 1-8 A method for detecting scoliosis based on depth cameras and infrared thermal imaging includes the following steps: Step S1: Acquire depth image data and infrared thermal imaging data of the subject's back; the depth image data is acquired by a depth camera at a preset shooting distance, and the infrared thermal imaging data is acquired simultaneously by an infrared thermal imager. Specifically, the first step is to set up and calibrate the equipment, including the following steps: (1) In this embodiment, the Intel RealSense D435i depth camera and the FLIR Lepton 3.5 infrared thermal imager are used as examples. The two are fixedly installed on the same aluminum alloy bracket, with their optical axes parallel and a horizontal distance of about 4 cm. (2) Stereo calibration was performed using a checkerboard calibration board, which covered the geometric range (visible light) that the depth camera could perceive and the thermal radiation range (heated checkerboard) that the infrared thermal imager could perceive. The intrinsic parameter matrices of the depth camera and the infrared thermal imager were obtained using the Zhang calibration method. The spatial transformation matrix R and translation vector T between the two cameras were obtained. The stereo matching algorithm included the PnP (Perspective-n-Point) + ICP (Iterative Closest Point) optimization algorithm.

[0025] Secondly, data collection and posture guidance are conducted, including the following steps: (1) The subject removes his / her upper garment and exposes his / her back (women may wear sports bras). He / She sits quietly for 5 minutes in an indoor environment with an ambient temperature of 22℃-26℃, no direct sunlight and no strong convective airflow, so that the body surface temperature can be stabilized. (2) Guide the subject to stand at the marked position 1.2 meters away from the collection device, with feet naturally separated shoulder-width apart, body relaxed and upright, arms hanging naturally, and eyes looking forward; (3) Start posture guidance function: The depth camera continuously captures real-time images and calculates the angle between the shoulder line and the horizontal plane; if the angle exceeds ±3°, the screen displays the shoulder offset direction and correction arrow, and the voice prompts "Please flatten your shoulders". When the posture meets the requirements for more than 1 second, synchronous acquisition is automatically triggered. (4) The synchronous trigger signal is sent by the host computer to the depth camera and the infrared thermal imager simultaneously through the GPIO hardware pin, controlling the two to start exposure synchronously, ensuring that the time deviation between the depth image frame and the infrared thermal image frame is less than 10 milliseconds; acquire a depth image with a resolution of 640×480 (16-bit depth value) and an infrared thermal image with a resolution of 320×240 (each pixel value is the temperature value, unit ℃).

[0026] Step S2: Preprocess the depth image data to generate a 3D point cloud of the back, and extract the surface morphology features of the back based on the 3D point cloud of the back. Preprocessing of depth image data to generate a 3D point cloud of the back side includes: (1) Denoising the depth image data is performed by using a bilateral filtering algorithm to remove artifacts and noise points in the depth image. Spatial domain variance =5, range variance =0.05, Used to control the size of the spatial neighborhood for filtering. A similarity threshold is used to control depth values; bilateral filtering can smooth noise while preserving edge information of the back contour. (2) Using the intrinsic parameter matrix of the depth camera ( , , , The denoised depth image is converted into a 3D point cloud. For a pixel with depth value d at pixel coordinates (u, v) in the depth image, its corresponding 3D spatial coordinates (X, Y, Z) are calculated using the following formula: ; ; ; in, , These are pixel coordinates; This is the depth value; , Main point; , These are the focal lengths along the X and Y axes, respectively.

[0027] By traversing each valid pixel (u, v, d) of the depth image, the three-dimensional spatial coordinates (X, Y, Z) are calculated, resulting in an original point cloud of approximately 300,000 points.

[0028] (3) A plane segmentation algorithm based on RANSAC (random sample consensus) is used to segment the largest plane (wall background) from the original 3D point cloud. The distance threshold is set to 1 cm, and points belonging to this plane are removed. Then, outliers are removed by statistical filtering (the average distance within 30 neighborhoods of each point is calculated, and points that exceed twice the standard deviation of the global average distance are removed).

[0029] (4) Perform voxel grid downsampling on the filtered back region point cloud, dividing the point cloud into voxel grids with a side length of 2mm to 5mm. In this embodiment, the voxel side length is 3mm. The centroid of each voxel is taken as the representative point to obtain a uniform density back three-dimensional point cloud. In this embodiment, the point cloud is simplified to about 8000-15000 points.

[0030] Based on the 3D point cloud of the back, the surface topography features of the back are extracted, specifically including: (1) Compute the curvature features of the back surface from three-dimensional points on the back, including Gaussian curvature and mean curvature; specifically, calculate the principal curvatures k1 and k2 of the local surface at each point, and then calculate the Gaussian curvature K=k1×k2 and the mean curvature H=(k1+k2) / 2. The Gaussian curvature is used to locate high-curvature anatomical landmarks such as the scapula and spinal groove.

[0031] (2) Extracting the midsagittal line of the back using a symmetry analysis-based method: In this embodiment, the projection of the back point cloud onto the coronal plane (YZ plane) can be used to register the left and right half-body point clouds by iterative nearest point (ICP). The intersection of the symmetry plane obtained by registration with the back surface is then used to extract the midsagittal line. The back is divided into a left-side region and a right-side region based on the midsagittal line. As another optional implementation method, the point cloud is projected onto the YZ plane to construct a symmetry evaluation function. ,in The rotation angle is the axis of symmetry. Let E be the offset distance. Grid search and gradient descent optimization are used to find the value that minimizes E. This yields the axis of symmetry. The intersection of the spatial plane corresponding to this axis of symmetry and the back point cloud is the median sagittal.

[0032] (3) Divide the back point cloud into the left point cloud. and the point cloud on the right Calculate the surface morphology difference index between the left and right regions, including: Average surface height difference between the left and right sides The calculation formula is: ,in, and These are the heights of the corresponding points on the left and right sides, respectively, which are the values ​​along the depth direction of the Z-axis; Maximum height difference between the left and right sides The calculation formula is: ; Asymmetry of projected area between left and right sides The calculation formula is: ;in, This represents the projected area on the left side of the back. This represents the projected area on the right side of the back.

[0033] (4) Extract the relative height features and rotation angle features of the scapular region. Locate the scapular region by the curvature extreme point. In this embodiment, the scapular region is located by searching for the cluster center of the local maximum value of Gaussian curvature. Calculate the vertical height difference of the YZ coronal plane of the left and right scapulae and the rotation angle difference relative to the midline sagittal line.

[0034] Step S3: Preprocess the infrared thermal imaging data, extract the temperature distribution characteristics on both sides of the back, and calculate the temperature asymmetry index on both sides. Extracting the bilateral temperature distribution features of the back specifically includes: (1) Based on the pre-calibrated spatial transformation matrix (R, T) between the depth camera and the infrared thermal imager, the temperature value of each pixel in the infrared thermal image is mapped to the three-dimensional coordinate system of the depth camera through perspective projection. For each pixel of the infrared thermal image ( The corresponding three-dimensional ray is obtained by back projection, and the intersection point is found with the three-dimensional point cloud surface of the back. The nearest intersection point is assigned the temperature value of the pixel, and finally a colored point cloud with temperature label is obtained for each three-dimensional point of the back.

[0035] (2) On the registered back 3D point cloud, the left temperature region and the right temperature region are divided with the median sagittal line as the reference. Since the depth camera and the infrared thermal imager have completed spatial registration, this division is consistent with the geometric region division in step S2. (3) Calculate the statistical temperature characteristics of the left and right temperature regions respectively, including: the average temperature of the left side. and the average temperature on the right Standard deviation of temperature on the left side and the standard deviation of temperature on the right Maximum temperature on the left side and the maximum temperature on the right Minimum temperature on the left and the minimum temperature on the right In this embodiment, = 32.1℃, =32.8℃, with a temperature difference of 0.7℃ between the two sides.

[0036] (4) The back is divided into multiple transverse bands along the midsagittal line (i.e., the direction of the spine, Y-axis, from C7 to L5 level), preferably 5-10 bands. In this embodiment, it is divided into 8 bands. The temperature gradient values ​​of the left and right sides in each band are calculated. It is found that the temperature difference is the largest in the thoracic segment (bands 2-5), followed by the lumbar segment (bands 6-8), and the smallest in the cervical segment (band 1). This distribution feature is highly consistent with the asymmetrical distribution pattern of paraspinal muscle tension in scoliosis patients.

[0037] The two-sided temperature asymmetry index is calculated as follows: Constructing a two-sided temperature asymmetry index The calculation formula is: ; in, The number of strip-shaped regions, For the first Average temperature on the left side of the strip-shaped region; For the first Average temperature on the right side of the strip-shaped region; The value range is [0, 1]; the larger the value, the more significant the temperature asymmetry on both sides.

[0038] In this embodiment, the examinee =0.023.

[0039] Simultaneously calculate the entropy of the temperature difference between the two sides. The calculation formula is: ; in, For the first The absolute value of the temperature difference between corresponding points on the left and right sides of each pixel; This represents the normalized probability density of the temperature difference distribution; To prevent small constants from taking the value of 0 in logarithmic operations; This represents the number of intervals in the temperature difference histogram.

[0040] Construct a temperature difference histogram (50 intervals) with an interval width of 0.1℃, and calculate the entropy of the temperature difference on both sides. In this embodiment, the examinee = 1.86.

[0041] Step S4: Perform multimodal feature fusion of the back surface morphology features and the bilateral temperature asymmetry index to construct a fused feature vector; Multimodal feature fusion specifically includes: (1) Construct the first feature vector (Depth morphological features), including: mean surface height difference Maximum height difference Area asymmetry ratio Relative height of the left scapula Relative height of the right scapula Difference in rotation angle of the scapula relative to the midline ; ; is a 6-dimensional feature vector.

[0042] (2) Construct the second feature vector (Infrared thermal characteristics), including: indicators of bilateral temperature asymmetry Two-sided temperature difference entropy Local temperature difference in each zone; ; represents a 10-dimensional feature vector, where, This represents the temperature difference between the left and right sides of the first strip region, and so on.

[0043] (3) To and Z-score normalization was performed to ensure a mean of 0 and a standard deviation of 1, thus eliminating the influence of different physical dimensions on the fusion process. After normalization... ; .

[0044] (4) Employing a multi-head attention mechanism for and Cross-attention calculation is performed. In this embodiment, a 2-head attention mechanism is used, with each head having a dimension of 32. As a query projection As projections of the key and value, compute the scaled dot product attention: ; in, For query vector; The key vector; A vector of values; It is a single-head feature dimension; Calculation for transpose; This is the weight normalization function.

[0045] At the same time As a query vector The inverse attention weights are calculated using the key vector and value vector. After concatenating the attention outputs from both directions, the result is normalized and passed through a fully connected layer to obtain a 32-dimensional fused feature vector. The introduction of this attention mechanism enables the model to adaptively learn the contribution weights of different input modalities to classification decisions. For example, it automatically increases the weight of infrared thermal features when surface morphology differences are not significant, and vice versa.

[0046] Step S5: Input the fused feature vector into the preset scoliosis classification and detection model, and output the scoliosis detection result; Please see Figure 7 The diagram shows the structure of the scoliosis classification and detection model in this invention. The scoliosis classification and detection model is a multi-task classification and regression network based on the Transformer encoder architecture, and its structure includes: Input layer: Receives a 32-dimensional fused feature vector ; Location encoding layer: Adds learnable location encoding to the feature dimension; The Transformer encoder layers consist of 2-4 layers, each containing a multi-head self-attention sub-layer and a feedforward neural network sub-layer; where Add represents residual addition, Norm represents layer normalization, FFN represents feedforward network, GELU represents Gaussian error linear unit activation function, FC represents fully connected layer, and ReLU represents linear rectified activation function.

[0047] Classification Header: Outputs probability distributions for four types of scoliosis: normal, C-type thoracic, C-type lumbar, and S-type. In this embodiment, the classification head outputs four probability distributions: [0.02, 0.91, 0.05, 0.02]; that is, normal 2%, C-type thoracic 91%, C-type lumbar 5%, and S-type 2%. The predicted category is "C-type thoracic scoliosis".

[0048] Regression head: Outputs the estimated Cobb angle value. In this embodiment, the estimated Cobb angle value for the subject is 28.4°. Automatic severity level determination: A Cobb angle between 25° and 45° indicates moderate scoliosis.

[0049] The training process of the model includes: (1) Obtain training datasets, collect depth images and infrared thermography synchronous data of patients with scoliosis diagnosed by clinical X-ray and normal control groups, and have orthopedic professionals label the scoliosis type and Cobb angle value.

[0050] (2) For each training sample, extract the fusion feature vector according to the process described in steps S2-S4.

[0051] (3) Input the fused feature vector into the classification network to obtain the predicted classification result and the predicted Cobb angle value.

[0052] (4) The cross-entropy loss function is adopted as the classification loss: ; where, is the number of categories, is the ground-truth label, is the predicted probability.

[0053] (5) The Smooth L1 loss function is adopted as the regression loss to penalize the error of the predicted Cobb angle value, and the calculation formula is: ; where, is the predicted angle; is the ground-truth X-ray angle.

[0054] (6) The total loss function is the weighted sum of the classification loss and the regression loss: ; where, and are weight coefficients, with a preferred value range of ∈[0.3, 0.7], ∈[0.3, 0.7], and .

[0055] (7) The AdamW optimizer is used, with an initial learning rate of 1×10 -4 , combined with a cosine annealing learning rate scheduling strategy, early stopping is triggered when the loss on the validation set no longer decreases for 5 consecutive epochs, and the network parameters are updated through backpropagation until convergence.

[0056] Step S6: According to the scoliosis detection result, generate a detection report containing the scoliosis type, scoliosis severity assessment grade and recommended measures. The detection report includes: The ID of the subject, the detection date and time; The judgment result of the scoliosis type (normal / C-type thoracic segment / C-type lumbar segment / S-type scoliosis); in this embodiment, the result is C-type thoracic segment scoliosis; The estimated Cobb angle and the severity grading of scoliosis, where a Cobb angle <10° is normal, 10°≤Cobb angle≤25° is mild, 25°<Cobb angle≤45° is moderate, and Cobb angle>45° is severe; in this embodiment, the Cobb angle is 28.4°, with a 95% confidence interval of [25.1°, 31.7°]; The severity, which is moderate in this embodiment; A visualization of the three-dimensional topography of the back is presented in the form of a heat map, which visually shows the color distribution of the asymmetrical areas on both sides; the color from blue to red indicates the degree of asymmetry from slight to severe. A thermogram showing temperature asymmetry on both sides highlights areas of temperature abnormality. Recommended measures include "It is recommended to visit an orthopedic clinic", "It is recommended to have regular follow-up examinations (every 6 months)" or "No obvious scoliosis is observed, normal follow-up is recommended". In this embodiment, the recommended measure is "It is recommended to visit an orthopedic clinic".

[0057] Example 2, Detection System.

[0058] Please see Figure 9 The detection system of the present invention includes the following modules: The data acquisition module includes a depth camera and an infrared thermal imager. The depth camera is preferably a structured light depth camera (such as the Intel RealSense series) or a ToF (Time-of-Flight) depth camera, with a working distance of 0.8 meters to 2.0 meters and a depth resolution of at least 640×480. The infrared thermal imager has a thermal sensitivity NETD (Noise Equivalent Temperature Difference) of less than or equal to 50 mK, a temperature measurement range of 20℃ to 40℃, and a spatial resolution of at least 320×240. The depth camera and infrared thermal imager are fixedly mounted on the same bracket with a fixed relative pose, facilitating continuous use after a single calibration. In this embodiment, the depth camera is selected as Orbbec AstraPro (structured light solution, depth resolution 640×480@30fps), and the infrared thermal imager is selected as Hikvision DS-2TP31B (NETD≤40mK, 320×240). Both are mounted on a foldable tripod, weighing approximately 1.5 kg, making them easy to carry and deploy quickly. The acquisition module has a built-in embedded control board (NVIDIA Jetson Nano) responsible for synchronous trigger control and image data temporary storage.

[0059] The depth data processing module, running on the control board's GPU, performs operations such as depth map denoising, 3D point cloud generation, background segmentation, and feature extraction. This module uses CUDA acceleration, with a single frame processing time of approximately 200ms.

[0060] After receiving the spatial registration information from the depth module, the infrared data processing module maps the infrared temperature grid data to three-dimensional space and performs bilateral temperature region division and asymmetry index calculation.

[0061] The multimodal fusion module is the core innovative module of this system. It cross-fuses the feature vectors extracted from the depth modality and the infrared modality using an attention mechanism to construct a fused feature vector. The network structure parameters of this module are fixed in the ONNX model file and achieve efficient inference on Jetson Nano through the TensorRT inference engine (fusion time <10ms). The multimodal fusion module includes: The feature normalization unit is used to perform Z-score normalization on the dorsal surface morphology features and the bilateral temperature asymmetry index, respectively; the cross-attention calculation unit is used to perform bidirectional cross-attention calculation on the normalized depth features and infrared features using a multi-head attention mechanism. The feature concatenation unit is used to perform residual connection, layer normalization, and fully connected projection on the attention calculation results and the original features to generate a fused feature vector.

[0062] The classification and detection module, also based on the TensorRT inference engine, outputs both the classification result and the Cobb angle estimate after a single forward propagation following the fusion of feature vector inputs. Inference time is <5ms. The model has been trained using over 12,000 clinical samples (approximately 3,500 of which are positive for scoliosis).

[0063] The report generation module generates a report based on the scoliosis detection results, including the type of scoliosis, the assessment level of the degree of scoliosis, and recommended measures. It supports print output, and the report language can be switched between Chinese and English, making it suitable for international schools and overseas medical examination scenarios.

[0064] With user authorization, the cloud-based model update module allows the system to optionally upload anonymized detection data (containing only extracted feature vectors and final clinical diagnostic results, excluding the original images) to the cloud server. The cloud-based system periodically uses the new data to incrementally train and fine-tune the model, then distributes the updated model parameters to each terminal device, ensuring that the system's detection accuracy continuously improves with increased usage.

[0065] The posture guidance module is used to detect in real time whether the subject's posture meets the standard requirements using a depth camera before data acquisition, and outputs guidance prompts (such as voice prompts "Please relax and stand naturally" or posture correction animations on the screen) when it does not meet the standard requirements.

[0066] The data quality control module is used to evaluate the quality of the acquired depth image data and infrared thermal imaging data (such as checking the effective pixel ratio of the depth image, the contrast of the infrared image, etc.), and prompts for re-acquisition when the quality does not meet the preset threshold.

[0067] Example 3, test results.

[0068] Test Example 1: The present invention was evaluated on the following test set: Test set composition: The independent test set contains 1200 samples, including 600 normal controls, 250 cases of C-type thoracic scoliosis, 200 cases of C-type lumbar scoliosis, and 150 cases of S-type scoliosis. All samples were confirmed as the gold standard label by X-ray Cobb angle measurement. None of the samples were used for model training.

[0069] Comparison method: Method A: Using only 3D topographic features from a depth camera (single-modal); Method B: Using only infrared thermal imaging temperature features (single-mode); Method C: The method of this invention (depth + infrared multimodal fusion).

[0070] Please refer to Table 1 for the evaluation results.

[0071] Table 1: Evaluation Results

[0072] As shown in Table 1, the multimodal fusion scheme proposed in this invention outperforms the single-modal method in all evaluation metrics. Particularly noteworthy is that for mild scoliosis samples (87 cases) with a Cobb angle less than 15°, the detection sensitivity of the method in this invention reached 89.7%, higher than the 76.4% using only a depth camera and the 61.2% using only infrared thermal imaging. This fully demonstrates the supplementary detection value of infrared thermal imaging modality for early, mild scoliosis; that is, when surface morphology changes are not yet significant, the bilateral temperature difference caused by asymmetric metabolism of the paraspinal muscles provides crucial clues for detection.

[0073] Test 2: To verify the detection accuracy and repeatability of this system, ATR angle repeatability test and Cobb angle comparison test were performed on 11 subjects.

[0074] (1) Stability verification of ATR angle test; Each subject underwent three consecutive trunk rotation angle (ATR) measurements in the thoracic, thoracolumbar, and lumbar segments under identical conditions, and the mean and standard deviation were calculated. Taking subject 1 as an example, the three thoracic ATR measurements were -2.8°, -2.6°, and -2.1°, with a mean of -2.50° and a standard deviation of 0.36°; the three thoracolumbar ATR measurements were -1.8°, -1.7°, and -1.2°, with a mean of -1.57° and a standard deviation of 0.32°; and the three lumbar ATR measurements were -0.9°, -3.9°, and 0°, with a mean of -1.60° and a standard deviation of 2.04°. The standard deviation of repeated measurements for the 11 subjects ranged from 0.29° to 2.71°, indicating that the present invention has good measurement repeatability.

[0075] (2) Verification of the accuracy of the Cobb angle test; Using X-ray measurements as the gold standard, the Cobb angle measurements of the same subject by this invention and X-ray equipment were compared. The comparison results are shown in Table 2.

[0076] Table 2: Comparison Results

[0077] As shown in Table 2, the measurement deviation range of the four subjects was -1.99° to 2.91°, indicating that the present invention has good consistency with the X-ray gold standard, while avoiding the risk of ionizing radiation from X-ray examination.

[0078] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0079] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for detecting scoliosis based on depth camera and infrared thermal imaging, comprising the following steps: Step S1: Acquire depth image data and infrared thermal imaging data of the subject's back; the depth image data is acquired by a depth camera at a preset shooting distance, and the infrared thermal imaging data is acquired simultaneously by an infrared thermal imager. Step S2: Preprocess the depth image data to generate a three-dimensional point cloud of the back, and extract the surface morphology features of the back based on the three-dimensional point cloud of the back. Step S3: Preprocess the infrared thermal imaging data, extract the temperature distribution characteristics on both sides of the back, and calculate the temperature asymmetry index on both sides. Step S4: Perform multimodal feature fusion of the back surface morphology features and the bilateral temperature asymmetry index to construct a fused feature vector; Multimodal feature fusion employs a bidirectional cross-attention mechanism, specifically including: constructing a first feature vector containing back surface topography features. and the second eigenvector containing the two-sided temperature asymmetry index ,right and Perform Z-score normalization separately; As a query vector Attention weights are calculated using the key vector and value vector, while... As a query vector The inverse attention weights are calculated using the key and value vectors, and the bidirectional attention output is residually connected to the original features and then normalized. Finally, the fused feature vector is obtained by projection through a fully connected layer. ; Step S5: Input the fused feature vector into the preset scoliosis classification and detection model, and output the scoliosis detection result; Step S6: Based on the scoliosis detection results, generate a detection report that includes the type of scoliosis, the assessment level of the degree of scoliosis, and recommended measures.

2. The scoliosis detection method based on depth camera and infrared thermal imaging according to claim 1, characterized in that: In step S1, synchronous acquisition specifically involves: pre-calibrating the spatial transformation matrix between the depth camera and the infrared thermal imager, and controlling the synchronous exposure of the depth camera and the infrared thermal imager by a synchronous trigger signal during acquisition.

3. The scoliosis detection method based on depth camera and infrared thermal imaging according to claim 1, characterized in that: In step S2, the depth image data is preprocessed to generate a three-dimensional point cloud of the back, specifically including: Denoising processing is performed on the depth image data, and a bilateral filtering algorithm is used to remove artifacts and noise points in the depth image; The denoised depth image is converted into a 3D point cloud using the intrinsic parameter matrix of the depth camera; A RANSAC-based planar segmentation algorithm is used to segment and remove the background plane from the 3D point cloud, while retaining the point cloud of the back region. Voxel downsampling was performed on the point cloud of the back region to obtain a uniform density 3D point cloud of the back.

4. The scoliosis detection method based on depth camera and infrared thermal imaging according to claim 1, characterized in that: In step S2, extracting the surface topography features of the back based on the three-dimensional point cloud of the back specifically includes: The curvature features of the back surface, including Gaussian curvature and mean curvature, are calculated from three-dimensional points on the back. The midsagittal line of the back was extracted using a symmetry-based analysis method, and the back was divided into left and right regions based on the midsagittal line. Calculate the surface morphology difference index between the left and right regions, including: the mean surface height difference between the left and right sides. The maximum height difference between the left and right sides And the asymmetry ratio of the projected areas on the left and right sides. ; The relative height and rotation angle features of the scapular region are extracted. The scapular region is located by the curvature extremum point. The vertical height difference between the left and right scapulae in the YZ coronal plane and the rotation angle difference relative to the midline sagittal line are calculated.

5. The scoliosis detection method based on depth camera and infrared thermal imaging according to claim 1, characterized in that: In step S3, extracting the temperature distribution features on both sides of the back specifically includes: Based on the pre-calibrated spatial transformation matrix between the depth camera and the infrared thermal imager, the acquired two-dimensional temperature data is mapped to the three-dimensional point cloud space on the back, so that each three-dimensional surface point on the back has a corresponding temperature attribute value. On the registered back 3D point cloud, the left and right temperature regions are divided with the median sagittal line as the reference. Calculate the statistical temperature characteristics of the left and right temperature regions respectively, including: the average temperature of the left side. and the average temperature on the right Standard deviation of temperature on the left side and the standard deviation of temperature on the right Maximum temperature on the left side and the maximum temperature on the right Minimum temperature on the left and the minimum temperature on the right ; The back is divided into multiple transverse bands along the midsagittal line, and the temperature gradient values ​​on the left and right sides of each band are calculated.

6. The scoliosis detection method based on depth camera and infrared thermal imaging according to claim 1, characterized in that: In step S3, the bilateral temperature asymmetry index is calculated, specifically as follows: Constructing a two-sided temperature asymmetry index The calculation formula is: ； in, The number of strip-shaped regions, For the first Average temperature on the left side of the strip-shaped region; For the first Average temperature on the right side of the strip-shaped region; The value range is [0, 1]; Simultaneously calculate the entropy of the temperature difference between the two sides. The calculation formula is: ； in, For the first The absolute value of the temperature difference between corresponding points on the left and right sides of each pixel; This represents the normalized probability density of the temperature difference distribution; To prevent small constants from taking the value of 0 in logarithmic operations; This represents the number of intervals in the temperature difference histogram.

7. The scoliosis detection method based on depth camera and infrared thermal imaging according to claim 1, characterized in that: In step S4, the first feature vector Includes: average surface height difference Maximum height difference Area asymmetry ratio Relative height of the left scapula Relative height of the right scapula Difference in rotation angle of the scapula relative to the midline ; Second eigenvector Includes: bilateral temperature asymmetry index Two-sided temperature difference entropy Local temperature difference in each zone; right and Z-score normalization was performed to make the mean 0 and the standard deviation 1.

8. The scoliosis detection method based on depth camera and infrared thermal imaging according to claim 1, characterized in that: In step S5, the scoliosis classification and detection model is a multi-task classification and regression network based on the Transformer encoder architecture, and its structure includes: Input layer: Receives the fused feature vector ; Location encoding layer: Adds learnable location encoding to the feature dimension; 2-4 Transformer encoder layers, each containing a multi-head self-attention sublayer and a feedforward neural network sublayer; Classification Header: Outputs the probability distribution of four types of scoliosis: normal, C-type thoracic, C-type lumbar, and S-type. Regression head: Outputs the estimated Cobb angle.

9. The scoliosis detection method based on depth camera and infrared thermal imaging according to claim 8, characterized in that: The model is trained using classification loss. and regression loss The weighted sum is used as the total loss function, and the calculation formula is as follows: ； in, and These are the weighting coefficients, and The classification loss uses the cross-entropy loss function, and the regression loss uses the smoothing L1 loss function.

10. A scoliosis detection system based on a depth camera and infrared thermal imaging, characterized in that, include: The data acquisition module, including a depth camera and an infrared thermal imager, is used to simultaneously acquire depth image data and infrared thermal imaging data of the subject's back. The depth data processing module is used to preprocess the depth image data to generate a three-dimensional point cloud of the back and extract the surface morphology features of the back. The infrared data processing module is used to preprocess the infrared thermal imaging data, extract the temperature distribution characteristics on both sides of the back, and calculate the temperature asymmetry index on both sides. A multimodal fusion module is used to fuse the back surface topography features with the bilateral temperature asymmetry index to construct a fused feature vector. The classification and detection module has a built-in trained scoliosis classification and detection model, which is used to output scoliosis detection results based on the fused feature vector. The report generation module is used to generate a test report based on the scoliosis test results, which includes the type of scoliosis, the assessment level of the degree of scoliosis, and recommended measures.

Images