A multi-modal image-based scoliosis rapid screening system and method

Abstract

The application belongs to the technical field of biomedical engineering, and discloses a kind of based on multimodal image's scoliosis rapid screening system, including following module: multimodal image acquisition module, data preprocessing module, back region segmentation module, posture evaluation and guiding module, dynamic midline and ATR calculation module, quality monitoring and output module.Integrating multimodal image acquisition, preprocessing, segmentation, posture evaluation and ATR calculation into a complete technical solution, the problem of inaccurate measurement, great influence of environmental light and individual posture, and dependence on manual operation in existing scoliosis screening technology is solved.Through the multimodal fusion strategy of depth image, color image and infrared image, the geometric characteristics of depth image and the penetration of infrared image are fully utilized, the interference of clothing texture and color on the screening result is significantly reduced, the dressing requirements for the screening object are greatly reduced, and the screening accuracy can still be guaranteed under non-ideal dressing conditions.

Description

Technical Field

[0001] This invention belongs to the field of biomedical engineering technology, specifically a rapid scoliosis screening system and method based on multimodal images. Background Technology

[0002] Scoliosis is a three-dimensional deformity of the spine, referring to the lateral curvature of the spine in the coronal plane, which deviates from the midline. It is often accompanied by vertebral rotation and changes in the physiological curvature in the sagittal plane, rather than a simple left-right curvature. There are many clinical subtypes of this disease, among which idiopathic scoliosis is the most common, and it is more common in adolescents. Congenital scoliosis is related to abnormal spinal development, while neuromuscular scoliosis is often secondary to diseases such as cerebral palsy and muscular dystrophy.

[0003] Scoliosis is a common spinal deformity, and early screening is crucial for prevention and treatment. Currently, there are various scoliosis screening technologies in clinical practice, but existing technologies still have many shortcomings. Traditional screening methods, such as X-ray imaging, pose radiation risks and rely on professional physicians, resulting in high costs and low efficiency. Existing vision-based screening methods usually have strict requirements on the examinee's clothing and posture, such as requiring an exposed back or wearing tight clothing, which is difficult to achieve in large-scale screening. At the same time, their robustness in complex environments such as changes in lighting and occlusion is insufficient, leading to a decrease in screening accuracy. In addition, existing technologies generally suffer from inaccurate measurements, are greatly affected by ambient lighting and individual posture, and rely on manual operation, failing to meet the needs of rapid screening in large numbers. Therefore, improvements are needed. Summary of the Invention

[0004] The purpose of this invention is to provide a rapid scoliosis screening system and method based on multimodal images to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a rapid scoliosis screening system based on multimodal images, comprising the following modules: a multimodal image acquisition module, a data preprocessing module, a back region segmentation module, a posture assessment and guidance module, a dynamic midline and ATR calculation module, and a quality monitoring and output module; The multimodal image acquisition module uses a depth camera, a color camera, and an infrared sensor based on the time-of-flight principle to simultaneously acquire depth images, color images, and infrared images of the subject's back. The data preprocessing module performs calibration, noise filtering, and hole filling on the original acquired multimodal images to eliminate image interference factors; The back region segmentation module employs a cascaded segmentation algorithm based on multimodal feature fusion, which accurately extracts the back region by combining coarse segmentation, fine segmentation, and multi-scale optimization. The posture assessment and guidance module is based on a deep learning-based 3D posture estimation model, which detects key points of the human spine and assesses the subject's posture compliance, while providing real-time visual / voice feedback to guide adjustments. The dynamic midline and ATR calculation module constructs a local midline of the longitudinal profile of the back through a dynamic midline algorithm, calculates the ATR (trunk rotation angle) based on the depth difference and spatial distance between the left and right back regions, and classifies the risk level of scoliosis according to the ATR value. The quality monitoring and output module monitors the screening quality in real time based on a weighted scoring model of image quality, pose compliance, and segmentation accuracy, and generates a screening report and medical recommendations that include risk assessment results.

[0006] Preferably, the depth camera, color camera, and infrared sensor of the multimodal image acquisition module maintain a synchronized acquisition sequence to ensure accurate matching of the spatial position and time dimension of the three types of images; the acquisition device supports adaptive ambient light adjustment and can stably acquire effective image data in both strong light and weak light environments, adapting to different screening scenarios.

[0007] Preferably, the calibration operation of the data preprocessing module includes camera intrinsic parameter calibration and multimodal image registration to eliminate equipment errors and image misalignment problems; noise filtering adopts an adaptive median filtering algorithm to specifically filter noise types in different modal images; hole filling is based on the neighborhood pixel interpolation method to fill missing pixels in the depth image and ensure the integrity of the back contour.

[0008] Preferably, the back region segmentation module includes a coarse segmentation unit, a fine segmentation unit, and a multi-scale optimization unit; the coarse segmentation unit quickly locates the approximate back region based on region growth criteria of depth gradient and color similarity. in For depth gradient, For color values, The color value of the seed point. and An adaptive threshold; The fine segmentation unit uses an active contour model to optimize the boundary and energy function. Defined as: in For contour smoothing, For shape priors based on anatomical atlases, For the boundary attraction term, The differential arc length of the profile; The multi-scale optimization unit integrates multi-resolution segmentation results through an image pyramid, reducing boundary localization errors. The calculation is as follows: in For the first Scale boundary points, As a reference boundary, For adaptive weights.

[0009] Preferably, the posture assessment and guidance module includes a key point detection unit, a posture compliance assessment unit, and a feedback guidance unit; the key point detection unit regresses three-dimensional key points of the human body, such as the spinous process of the spine and the scapula, using a convolutional neural network, and the three-dimensional coordinates... Calculated using the following formula: in This is an attitude correction term based on infrared data; The posture compliance assessment unit sets standard standing posture parameter thresholds to determine whether the examinee's posture meets the screening requirements; the feedback guidance unit provides real-time adjustment guidance to examinees with non-compliant postures through a display device and a voice module.

[0010] Preferably, the dynamic centerline and ATR calculation module includes a dynamic centerline construction unit, an ATR calculation unit, and a risk level classification unit; the dynamic centerline construction unit determines the centerline point in the longitudinal section of the back side through curvature extreme point detection and symmetry analysis, and forms a dynamic centerline by spline curve fitting, with curvature... The calculation is as follows: in For depth value, For spatial coordinate axes; The ATR calculation unit is based on the depth difference between the left and right sampling regions. Spatial distance L, ATR angle The calculation is as follows: Multi-scale sliding window weighted fusion is used to improve the robustness of the results; The risk level classification unit is determined according to clinical medical standards: The risk level is low; if negative, regular follow-up is recommended. For those with a medium-risk suspected positive result, further medical clinical examination or X-ray-assisted diagnosis is recommended. The result is high risk and positive. The system generates an early warning report and strongly recommends that you go to a specialized hospital for spinal imaging examination.

[0011] Preferably, the quality scoring model of the quality monitoring and output module is as follows: in For image quality, Posture compliance score To score the accuracy of segmentation, Adjust dynamically according to the screening scenario; The quality monitoring and output module supports the instant printing and electronic document export of screening reports. The reports include image data, ATR values, risk levels, and targeted medical recommendations.

[0012] Preferably, the rapid scoliosis screening system employs a multimodal feature fusion strategy to improve screening accuracy, weightedly fusing features from depth images, color images, and infrared images using the following fusion formula: in, , , These are depth, color, and infrared feature vectors, respectively; weighting coefficients. , , Dynamic adjustment based on feature saliency map: in, This is a feature significance map, where feature significance is calculated using gradient magnitude and curvature; Feature saliency map The calculation integrates the spatial distribution characteristics of images from various modalities. The specific steps are as follows: Gradient magnitude calculation: Calculate the partial derivatives in the horizontal and vertical directions for each modality image to obtain the gradient magnitude. , used to characterize the edge strength of an image; Curvature Calculation: Calculating the average curvature of a local surface based on the second derivative. , used to characterize the geometric abrupt changes in the contour of the back muscles; Saliency fusion: The saliency plot is derived from a weighted average of gradient magnitude and curvature. in, and The normalized components; Used to control the proportion of contribution of image grayscale / color change intensity to saliency; Used to control the contribution ratio of abrupt changes in three-dimensional geometry to significance; In low light conditions, the infrared mode Gradient weights are automatically boosted, thereby... The addition of infrared penetration features makes the model more reliant on them, reducing clothing interference.

[0013] To achieve the aforementioned other objective, the present invention provides the following technical solution: a rapid scoliosis screening method based on multimodal images, which is applied to the aforementioned rapid scoliosis screening system based on multimodal images, and includes the following steps: S1, Multimodal Image Acquisition The screening equipment is set up, and the examinee stands in the designated area with his back to the acquisition module. The system simultaneously acquires back depth images, color images and infrared images through a depth camera, a color camera and an infrared sensor to ensure that the equipment and the examinee are in relatively fixed positions during the acquisition process. S2, Data Preprocessing The system automatically performs calibration operations on the acquired raw images to eliminate device intrinsic parameter errors and multimodal image registration deviations; it uses an adaptive median filtering algorithm to filter image noise and fills holes in the depth image using neighborhood pixel interpolation to output high-quality preprocessed images. S3, Back Region Segmentation First, a coarse segmentation is performed using a region growing algorithm, and the approximate area of ​​the back is quickly located based on depth gradient and color similarity criteria. Then, a fine segmentation is performed using an active contour model to optimize the boundary of the back region. Finally, the multi-resolution segmentation results are integrated using an image pyramid to obtain an accurate back region segmentation map. S4. Posture Assessment and Guidance The three-dimensional spinal key points of the examinee are extracted by convolutional neural network, the actual posture parameters are calculated and compared with standard thresholds to assess the compliance of the posture; if the posture is not compliant, the examinee is guided to adjust the standing posture through visual display and voice prompts until the screening requirements are met. S5, Dynamic Midline Construction and ATR Calculation In the preprocessed depth image, the curvature extrema of the longitudinal profile of the back are detected, and the midline point is determined by combining symmetry analysis and fitted as a dynamic centerline; the depth difference between the left and right back regions is calculated. Spatial distance The ATR angle is obtained through multi-scale sliding window weighted fusion. ,in accordance with The values ​​are categorized into low, medium, and high risk levels; S6. Quality Monitoring and Report Output A comprehensive quality score Q is calculated based on image quality, pose compliance, and segmentation accuracy. If Q meets the threshold requirements, the screening result is confirmed as valid, and a screening report containing image data, ATR value, risk level, and medical advice is generated. If Q does not meet the standard, the process returns to S1 for re-collection to ensure the reliability of the screening result.

[0014] The beneficial effects of this invention are as follows: By integrating multimodal image acquisition, preprocessing, segmentation, pose assessment, and ATR calculation into a complete technical solution, this approach effectively solves the problems of inaccurate measurements, significant influence from ambient lighting and individual posture, and reliance on manual operation in existing scoliosis screening technologies. Specifically, the multimodal fusion strategy using depth, color, and infrared images fully leverages the geometric characteristics of depth images and the penetrability of infrared images, significantly reducing the interference of clothing texture and color on screening results and greatly lowering the requirements for clothing on the screening subjects, while still ensuring screening accuracy under non-ideal clothing conditions. Furthermore, the cascaded processing architecture and parallel computing technology comprehensively optimize the algorithm flow. The optimization significantly shortens the processing time for a single screening case, enabling rapid screening of scoliosis in large batches. It is perfectly adapted to high-throughput screening scenarios such as schools and communities. Furthermore, through deep learning-based 3D posture assessment and real-time guidance, it effectively avoids the impact of individual posture deviations on screening results. Combined with multi-indicator quality monitoring and closed-loop optimization design, it further improves the reliability and stability of measurement results. The overall system has a high degree of automation, significantly reducing reliance on manual operation. Standardized screening can be carried out without professional medical personnel, expanding the screening coverage and providing efficient, convenient, and accurate technical support for the early detection and timely intervention of scoliosis. Attached Figure Description

[0015] Figure 1 This is a system block diagram of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] like Figure 1 As shown, this embodiment of the invention provides a rapid scoliosis screening system based on multimodal images, including the following modules: multimodal image acquisition module, data preprocessing module, back region segmentation module, posture assessment and guidance module, dynamic midline and ATR calculation module, and quality monitoring and output module; The multimodal image acquisition module uses a depth camera, a color camera, and an infrared sensor based on the time-of-flight principle to simultaneously acquire depth, color, and infrared images of the subject's back. The data preprocessing module performs calibration, noise filtering, and hole filling on the raw acquired multimodal images to eliminate image interference factors; The back region segmentation module adopts a cascaded segmentation algorithm based on multimodal feature fusion, which accurately extracts the back region by combining coarse segmentation, fine segmentation and multi-scale optimization. The posture assessment and guidance module is based on a deep learning-based 3D posture estimation model. It detects key points of the human spine and assesses the subject's posture compliance, while providing real-time visual / voice feedback to guide adjustments. The dynamic midline and ATR calculation module constructs a local midline of the longitudinal profile of the back through a dynamic midline algorithm, calculates the ATR (trunk rotation angle) based on the depth difference and spatial distance between the left and right back regions, and classifies the risk level of scoliosis according to the ATR value. The quality monitoring and output module monitors the screening quality in real time based on a weighted scoring model of image quality, pose compliance, and segmentation accuracy, and generates screening reports and medical recommendations that include risk assessment results.

[0018] By integrating multimodal image acquisition, preprocessing, segmentation, posture assessment, ATR calculation, and quality monitoring modules, a closed-loop screening system is constructed to automate the entire process of scoliosis from data acquisition to risk assessment report output. This effectively reduces reliance on manual operation, meets the needs of large-scale rapid screening, and improves the integrity and efficiency of the screening process.

[0019] Among them, the depth camera, color camera and infrared sensor of the multimodal image acquisition module maintain synchronous acquisition sequence to ensure accurate matching of the spatial position and time dimension of the three types of images; the acquisition device supports adaptive ambient light adjustment, and can stably acquire effective image data in both strong light and weak light environments, adapting to different screening scenarios.

[0020] By ensuring accurate spatiotemporal matching of multimodal images, the consistency of image acquisition is improved. The adaptive illumination adjustment function enhances the stability of the system in complex environments such as strong light and weak light, ensuring that effective raw data can be obtained in different scenarios, laying a high-quality data foundation for subsequent processing and improving the system's environmental adaptability.

[0021] 3. A rapid scoliosis screening system based on multimodal images according to claim 1, characterized in that: the calibration operation of the data preprocessing module includes camera intrinsic parameter calibration and multimodal image registration to eliminate equipment errors and image misalignment problems; noise filtering adopts an adaptive median filtering algorithm to specifically filter noise types of different modal images; hole filling is based on the neighborhood pixel interpolation method to fill missing pixels in the depth image and ensure the integrity of the back contour.

[0022] Camera intrinsic parameter calibration and multimodal registration eliminate equipment errors and image misalignment. Adaptive median filtering specifically filters out noise of different modes. Neighborhood pixel interpolation fills in depth image holes, effectively improving the quality of the preprocessed image and ensuring the integrity of the back contour. This provides reliable data support for subsequent core steps such as region segmentation and pose assessment.

[0023] The back region segmentation module includes coarse segmentation units, fine segmentation units, and multi-scale optimization units. The coarse segmentation unit quickly locates the approximate back region based on region growth criteria using depth gradient and color similarity. in For depth gradient, For color values, The color value of the seed point. and An adaptive threshold; The fine segmentation unit uses an active contour model to optimize the boundary and energy function. Defined as: in For contour smoothing, For shape priors based on anatomical atlases, For the boundary attraction term, The differential arc length of the profile; The multi-scale optimization unit integrates multi-resolution segmentation results through image pyramids, reducing boundary localization errors. The calculation is as follows: in For the first Scale boundary points, As a reference boundary, For adaptive weights.

[0024] We employ coarse segmentation to quickly locate the approximate area of ​​the back, fine segmentation to optimize the boundary, and multi-scale optimization to integrate multi-resolution results. Through a cascading strategy, we balance segmentation efficiency and accuracy, reduce boundary positioning errors, and improve the accuracy and robustness of back region segmentation in complex scenarios.

[0025] The posture assessment and guidance module includes a key point detection unit, a posture compliance assessment unit, and a feedback guidance unit. The key point detection unit uses a convolutional neural network to regress three-dimensional key points of the human body, such as the spinous processes of the spine and the scapula, and their three-dimensional coordinates. Calculated using the following formula: in This is an attitude correction term based on infrared data; The posture compliance assessment unit sets standard standing posture parameter thresholds to determine whether the examinee's posture meets the screening requirements; the feedback guidance unit provides real-time adjustment guidance to examinees with non-compliant postures through display devices and voice modules.

[0026] By accurately regressing key points of the spine using CNN and combining infrared data to correct three-dimensional posture, a scientific assessment of posture compliance is achieved. Real-time visual / voice feedback guides examinees to adjust their posture, effectively reducing the interference of non-compliant posture on screening results, improving screening accuracy, and optimizing the examinee's screening experience.

[0027] The dynamic centerline and ATR calculation module includes a dynamic centerline construction unit, an ATR calculation unit, and a risk level classification unit. Within the longitudinal profile of the back section, the dynamic centerline construction unit determines the centerline point through curvature extreme point detection and symmetry analysis, and forms the dynamic centerline using spline curve fitting. The calculation is as follows: in For depth value, For spatial coordinate axes; The ATR calculation unit is based on the depth difference between the left and right sampling regions. Spatial distance L, ATR angle The calculation is as follows: Multi-scale sliding window weighted fusion is used to improve the robustness of the results; Risk level classification units are determined according to clinical medical standards: The risk level is low; if negative, regular follow-up is recommended. For those with a medium-risk suspected positive result, further medical clinical examination or X-ray-assisted diagnosis is recommended. The result is high risk and positive. The system generates an early warning report and strongly recommends that you go to a specialized hospital for spinal imaging examination.

[0028] By constructing an accurate dynamic midline through curvature extremum point detection and spline curve fitting, and improving the robustness of ATR calculation through multi-scale sliding window weighted fusion, the risk level is classified according to clinical medical standards, thereby achieving accurate quantitative assessment of scoliosis risk, providing clear medical guidance for examinees, and assisting in early intervention and treatment.

[0029] The quality scoring model for the quality monitoring and output module is as follows: in For image quality, Posture compliance score To score the accuracy of segmentation, Adjust dynamically according to the screening scenario; The quality monitoring and output module supports instant printing and electronic document export of screening reports. The reports include image data, ATR values, risk levels, and targeted medical recommendations.

[0030] The multi-indicator weighted quality scoring model enables comprehensive monitoring of screening quality, dynamically adjusts weights to adapt to different screening scenarios, ensures reliable output results, and enhances the convenience of result presentation with instant report printing and electronic export functions, providing medical personnel and examinees with comprehensive and intuitive screening information.

[0031] The rapid scoliosis screening system employs a multimodal feature fusion strategy to improve screening accuracy. It weights and fuses features from depth images, color images, and infrared images using the following formula: in, , , These are depth, color, and infrared feature vectors, respectively; weighting coefficients. , , Dynamic adjustment based on feature saliency map: in, This is a feature significance map, where feature significance is calculated using gradient magnitude and curvature; Feature saliency map The calculation integrates the spatial distribution characteristics of images from various modalities. The specific steps are as follows: Gradient magnitude calculation: Calculate the partial derivatives in the horizontal and vertical directions for each modality image to obtain the gradient magnitude. , used to characterize the edge strength of an image; Curvature Calculation: Calculating the average curvature of a local surface based on the second derivative. , used to characterize the geometric abrupt changes in the contour of the back muscles; Saliency fusion: The saliency plot is derived from a weighted average of gradient magnitude and curvature. in, and The normalized components; Used to control the proportion of contribution of image grayscale / color change intensity to saliency; Used to control the contribution ratio of abrupt changes in three-dimensional geometry to significance; In low light conditions, the infrared mode Gradient weights are automatically boosted, thereby... The addition of infrared penetration features makes the model more reliant on them, reducing clothing interference.

[0032] Feature-level weighted fusion fully leverages the complementary advantages of depth, color, and infrared images, dynamically adjusts weights based on feature saliency maps, optimizes feature contributions in areas with complex boundaries, rich textures, and insufficient lighting, effectively reduces interference from factors such as clothing, lighting, and occlusion, and significantly improves screening accuracy and environmental robustness.

[0033] A rapid scoliosis screening method based on multimodal images, applied to the aforementioned rapid scoliosis screening system based on multimodal images, includes the following steps: S1, Multimodal Image Acquisition The screening equipment is set up, and the examinee stands in the designated area with his back to the acquisition module. The system simultaneously acquires back depth images, color images and infrared images through a depth camera, a color camera and an infrared sensor to ensure that the equipment and the examinee are in relatively fixed positions during the acquisition process. S2, Data Preprocessing The system automatically performs calibration operations on the acquired raw images to eliminate device intrinsic parameter errors and multimodal image registration deviations; it uses an adaptive median filtering algorithm to filter image noise and fills holes in the depth image using neighborhood pixel interpolation to output high-quality preprocessed images. S3, Back Region Segmentation First, a coarse segmentation is performed using a region growing algorithm, and the approximate area of ​​the back is quickly located based on depth gradient and color similarity criteria. Then, a fine segmentation is performed using an active contour model to optimize the boundary of the back region. Finally, the multi-resolution segmentation results are integrated using an image pyramid to obtain an accurate back region segmentation map. S4. Posture Assessment and Guidance The three-dimensional spinal key points of the examinee are extracted by convolutional neural network, the actual posture parameters are calculated and compared with standard thresholds to assess the compliance of the posture; if the posture is not compliant, the examinee is guided to adjust the standing posture through visual display and voice prompts until the screening requirements are met. S5, Dynamic Midline Construction and ATR Calculation In the preprocessed depth image, the curvature extrema of the longitudinal profile of the back are detected, and the midline point is determined by combining symmetry analysis and fitted as a dynamic centerline; the depth difference between the left and right back regions is calculated. Spatial distance The ATR angle is obtained through multi-scale sliding window weighted fusion. ,in accordance with The values ​​are categorized into low, medium, and high risk levels; S6. Quality Monitoring and Report Output A comprehensive quality score Q is calculated based on image quality, pose compliance, and segmentation accuracy. If Q meets the threshold requirements, the screening result is confirmed as valid, and a screening report containing image data, ATR value, risk level, and medical advice is generated. If Q does not meet the standard, the process returns to S1 for re-collection to ensure the reliability of the screening result.

[0034] By standardizing the entire process of multimodal image acquisition, data preprocessing, back region segmentation, posture assessment and guidance, dynamic midline construction and ATR calculation, quality monitoring and report output, the system achieves standardized and automated operation of scoliosis screening. Step S1 ensures spatiotemporal synchronization of multimodal images and stability of data acquisition; Step S2 optimizes data quality through precise calibration, targeted filtering and hole filling; Step S3's cascaded segmentation strategy balances efficiency and accuracy; Step S4's posture assessment and real-time guidance effectively avoid the interference of posture deviations on the results; Step S5's dynamic midline construction and multi-scale ATR calculation improve the accuracy of risk level classification; and Step S6's quality scoring and closed-loop resampling design strictly control the reliability of screening results. The overall process significantly shortens the screening time per case, reduces the complexity of manual operation and intervention costs, and not only meets the needs of large-scale high-throughput screening scenarios such as schools and communities, but also ensures the consistency and repeatability of screening results in different scenarios.

[0035] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0036] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A rapid scoliosis screening system based on multimodal images, characterized in that, It includes the following modules: multimodal image acquisition module, data preprocessing module, back region segmentation module, pose evaluation and guidance module, dynamic midline and ATR calculation module, and quality monitoring and output module; The multimodal image acquisition module uses a depth camera, a color camera, and an infrared sensor based on the time-of-flight principle to simultaneously acquire depth images, color images, and infrared images of the subject's back. The data preprocessing module performs calibration, noise filtering, and hole filling on the original acquired multimodal images to eliminate image interference factors; The back region segmentation module employs a cascaded segmentation algorithm based on multimodal feature fusion, which accurately extracts the back region by combining coarse segmentation, fine segmentation, and multi-scale optimization. The posture assessment and guidance module is based on a deep learning-based 3D posture estimation model, which detects key points of the human spine and assesses the subject's posture compliance, while providing real-time visual / voice feedback to guide adjustments. The dynamic midline and ATR calculation module constructs a local midline of the longitudinal profile of the back through a dynamic midline algorithm, calculates the ATR (trunk rotation angle) based on the depth difference and spatial distance between the left and right back regions, and classifies the risk level of scoliosis according to the ATR value. The quality monitoring and output module monitors the screening quality in real time based on a weighted scoring model of image quality, pose compliance, and segmentation accuracy, and generates a screening report and medical recommendations that include risk assessment results.

2. The rapid scoliosis screening system based on multimodal images according to claim 1, characterized in that: The depth camera, color camera, and infrared sensor of the multimodal image acquisition module maintain synchronized acquisition timing to ensure accurate matching of the spatial position and time dimension of the three types of images; the acquisition device supports adaptive ambient light adjustment and can stably acquire effective image data in both strong light and weak light environments, adapting to different screening scenarios.

3. The rapid scoliosis screening system based on multimodal images according to claim 1, characterized in that: The calibration operations of the data preprocessing module include camera intrinsic parameter calibration and multimodal image registration to eliminate equipment errors and image misalignment problems; noise filtering adopts an adaptive median filtering algorithm to specifically filter noise types in different modal images; hole filling is based on the neighborhood pixel interpolation method to fill missing pixels in the depth image and ensure the integrity of the back contour.

4. The rapid scoliosis screening system based on multimodal images according to claim 1, characterized in that: The back region segmentation module includes a coarse segmentation unit, a fine segmentation unit, and a multi-scale optimization unit; the coarse segmentation unit quickly locates the approximate back region based on region growth criteria of depth gradient and color similarity. in For depth gradient, For color values, The color value of the seed point. and An adaptive threshold; The fine segmentation unit uses an active contour model to optimize the boundary and energy function. Defined as: in For contour smoothing, For shape priors based on anatomical atlases, For the boundary attraction term, The differential arc length of the profile; The multi-scale optimization unit integrates multi-resolution segmentation results through an image pyramid, reducing boundary localization errors. The calculation is as follows: in For the first Scale boundary points, As a reference boundary, For adaptive weights.

5. A rapid scoliosis screening system based on multimodal images according to claim 1, characterized in that: The posture assessment and guidance module includes a key point detection unit, a posture compliance assessment unit, and a feedback guidance unit. The key point detection unit uses a convolutional neural network to regress three-dimensional key points of the human body, such as the spinous processes of the spine and the scapula, and their three-dimensional coordinates. Calculated using the following formula: in This is an attitude correction term based on infrared data; The posture compliance assessment unit sets standard standing posture parameter thresholds to determine whether the examinee's posture meets the screening requirements; the feedback guidance unit provides real-time adjustment guidance to examinees with non-compliant postures through a display device and a voice module.

6. The rapid scoliosis screening system based on multimodal images according to claim 1, characterized in that: The dynamic centerline and ATR calculation module includes a dynamic centerline construction unit, an ATR calculation unit, and a risk level classification unit. The dynamic centerline construction unit determines the centerline point within the longitudinal profile of the back section through curvature extreme point detection and symmetry analysis, and forms a dynamic centerline using spline curve fitting. The calculation is as follows: in For depth value, For spatial coordinate axes; The ATR calculation unit is based on the depth difference between the left and right sampling regions. Spatial distance L, ATR angle The calculation is as follows: Multi-scale sliding window weighted fusion is used to improve the robustness of the results; The risk level classification unit is determined according to clinical medical standards: The risk level is low; if negative, regular follow-up is recommended. For those with a medium-risk suspected positive result, further medical clinical examination or X-ray-assisted diagnosis is recommended. The result is high risk and positive. The system generates an early warning report and strongly recommends that you go to a specialized hospital for spinal imaging examination.

7. The rapid scoliosis screening system based on multimodal images according to claim 1, characterized in that: The quality scoring model of the quality monitoring and output module is as follows: in For image quality, Posture compliance score To score the accuracy of segmentation, Adjust dynamically according to the screening scenario; The quality monitoring and output module supports the instant printing and electronic document export of screening reports. The reports include image data, ATR values, risk levels, and targeted medical recommendations.

8. A rapid scoliosis screening system based on multimodal images according to claim 1, characterized in that: The rapid scoliosis screening system employs a multimodal feature fusion strategy to improve screening accuracy, weighting and fusing features from depth images, color images, and infrared images. The fusion formula is as follows: in, , , These are depth, color, and infrared feature vectors, respectively; weighting coefficients. , , Dynamic adjustment based on feature saliency map: in, This is a feature significance map, where feature significance is calculated using gradient magnitude and curvature; Feature saliency map The calculation integrates the spatial distribution characteristics of images from various modalities. The specific steps are as follows: Gradient magnitude calculation: Calculate the partial derivatives in the horizontal and vertical directions for each modality image to obtain the gradient magnitude. , used to characterize the edge strength of an image; Curvature Calculation: Calculating the average curvature of a local surface based on the second derivative. , used to characterize the geometric abrupt changes in the contour of the back muscles; Saliency fusion: The saliency plot is derived from a weighted average of gradient magnitude and curvature. in, and The normalized components; Used to control the proportion of contribution of image grayscale / color change intensity to saliency; Used to control the contribution ratio of abrupt changes in three-dimensional geometry to significance; In low light conditions, the infrared mode Gradient weights are automatically boosted, thereby... The addition of infrared penetration features makes the model more reliant on them, reducing clothing interference.

9. A rapid scoliosis screening method based on multimodal images, wherein the method is applied to the rapid scoliosis screening system based on multimodal images as described in any one of claims 1 to 8, characterized in that, Includes the following steps: S1, Multimodal Image Acquisition The screening equipment is set up, and the examinee stands in the designated area with his back to the acquisition module. The system simultaneously acquires back depth images, color images and infrared images through a depth camera, a color camera and an infrared sensor to ensure that the equipment and the examinee are in relatively fixed positions during the acquisition process. S2, Data Preprocessing The system automatically performs calibration operations on the acquired raw images to eliminate device intrinsic parameter errors and multimodal image registration deviations; it uses an adaptive median filtering algorithm to filter image noise and fills holes in the depth image using neighborhood pixel interpolation to output high-quality preprocessed images. S3, Back Region Segmentation First, a coarse segmentation is performed using a region growing algorithm, and the approximate area of ​​the back is quickly located based on depth gradient and color similarity criteria. Then, a fine segmentation is performed using an active contour model to optimize the boundary of the back region. Finally, the multi-resolution segmentation results are integrated using an image pyramid to obtain an accurate back region segmentation map. S4. Posture Assessment and Guidance The three-dimensional spinal key points of the examinee are extracted by convolutional neural network, the actual posture parameters are calculated and compared with standard thresholds to assess the compliance of the posture; if the posture is not compliant, the examinee is guided to adjust the standing posture through visual display and voice prompts until the screening requirements are met. S5, Dynamic Midline Construction and ATR Calculation In the preprocessed depth image, the curvature extrema of the longitudinal profile of the back are detected, and the midline point is determined by combining symmetry analysis and fitted as a dynamic centerline; the depth difference between the left and right back regions is calculated. Spatial distance The ATR angle is obtained through multi-scale sliding window weighted fusion. ,in accordance with The values ​​are categorized into low, medium, and high risk levels; S6. Quality Monitoring and Report Output A comprehensive quality score Q is calculated based on image quality, pose compliance, and segmentation accuracy. If Q meets the threshold requirements, the screening result is confirmed as valid, and a screening report containing image data, ATR value, risk level, and medical advice is generated. If Q does not meet the standard, the process returns to S1 for re-collection to ensure the reliability of the screening result.

Images