Scoliosis monitoring method based on multi-modal fusion system

Abstract

The invention provides a scoliosis monitoring method based on a multi-modal fusion system, and the method comprises the steps: collecting the foot three-dimensional ground reaction force and foot kinematics data through an intelligent insole integrated with a flexible force sensor and an inertial measurement unit, and carrying out the synchronous transmission through a communication unit. A human body posture video is obtained through a monocular camera, and posture parameters of the spine, the pelvis and the trunk are extracted through an improved human body key point detection model. And space-time fusion of kinematics data and video key point data is realized by adopting extended Kalman filtering, and multi-modal time sequence data is constructed. And generating a pseudo label based on a posture anomaly rule, establishing a multi-modal training data set containing the component data, the kinematics data, the posture parameters and the pseudo label, and training the scoliosis monitoring model through self-supervised pre-training and supervised fine tuning. And finally, outputting scoliosis evaluation indexes and abnormal risks, and realizing intelligent monitoring of the spine state.

Description

Technical Field

[0001] This invention relates to the field of biomechanical monitoring and rehabilitation assessment technology, and in particular to a non-invasive multimodal fusion monitoring system and method for scoliosis assessment. Background Technology

[0002] Scoliosis is a common spinal deformity, with a particularly high incidence among adolescents. Timely and accurate assessment of spinal condition is crucial for early screening, intervention, and monitoring of disease progression. Current methods for spinal biomechanical assessment mainly include laboratory-grade biomechanical analysis systems and portable wearable sensing devices.

[0003] Laboratory biomechanical spinal assessment systems typically include a force table and optical motion capture equipment. By measuring ground reaction force (GRF) and human kinematic parameters, they indirectly assess spinal load distribution and postural compensation characteristics. The specific components are as follows: (1) Force table system: It is fixedly installed on the laboratory floor and uses piezoelectric sensors to obtain the three-dimensional components of GRF such as vertical force (Fz), front and rear shear force (Fx), and left and right shear force (Fy); (2) Optical motion capture system: Kinematic data of the trunk and lower limbs are acquired by tracking optical markers and then spatiotemporally fused and analyzed with component data.

[0004] Such systems can quantify the correlation between GRF and spinal load during dynamic gait and calculate compensatory indices such as center of mass shift and pressure asymmetry index. However, they have significant drawbacks: 1) Scenario limitations: The force measuring platform can only capture single-step data. Due to the limited laboratory space, it cannot continuously collect data during long-distance (>100 steps) natural walking. 2) High cost and complexity: The system deployment relies on a professional laboratory environment, the equipment is expensive and the operation is complicated, making it unsuitable for daily use.

[0005] In addition, to improve scene adaptability, portable wearable devices are gradually being introduced into spinal posture monitoring. Common solutions include force-sensitive resistor (FSR) insoles and inertial measurement unit (2) devices: (1) The FSR insole has 8–16 pressure sensing units on the sole of the foot, which transmit pressure distribution data in real time via Bluetooth or wireless communication; the FSR insole can detect asymmetry in sole pressure. (2) The IMU device can acquire data such as trunk angular velocity, acceleration, and attitude angle, which can be used to analyze posture stability and degree of deviation, and to evaluate posture stability.

[0006] The aforementioned devices are easy to deploy in everyday scenarios and have a certain degree of posture recognition capability, but they have the following problems: Modal dissection: FSR insoles can only acquire vertical force parameters and cannot capture shear force (Fx / Fy) and motion parameters; One-sided information: Although IMU devices can sense posture, they lack a structured representation of plantar load, making it difficult to reconstruct the complete force transmission path, resulting in a one-sided biomechanical assessment; (3) Strong dependence on annotation: The training of machine learning models heavily depends on high-precision labeled data, while the existing annotation process is highly dependent on laboratory systems, which is costly and inefficient.

[0007] In summary, the shortcomings of the existing technology are as follows: (1) Poor scene adaptability: Due to limitations of the laboratory environment or insufficient sensor data modes, it is impossible to achieve long-term, multi-modal continuous monitoring of spinal load in natural walking scenarios; (2) Weak data fusion and modeling capabilities: Existing solutions have not achieved spatiotemporal alignment and joint modeling of GRF-motion-posture, making it difficult to accurately analyze the real transmission path of spinal mechanical load; (3) High cost and complicated operation: The force measuring table and other system equipment are expensive and complicated to deploy, making it difficult to promote and apply in daily environments; (4) Heavy reliance on annotation resources: A large number of training samples need to be manually annotated in the laboratory, which restricts the generalization ability and sustainable iteration of intelligent analysis models.

[0008] In summary, current technologies struggle to simultaneously meet the comprehensive requirements of scoliosis assessment in terms of scene adaptability, multimodal fusion capabilities, annotation efficiency, and monitoring continuity. Therefore, there is an urgent need for a technical solution that can achieve multimodal, long-term, biomechanical monitoring in natural settings, while also being low-cost and highly deployable. Summary of the Invention

[0009] The purpose of this invention is to provide a non-invasive, multimodal fusion spinal monitoring system, and a scoliosis monitoring method based on a multimodal fusion system combined with a self-supervised learning strategy.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: A method for monitoring scoliosis based on a multimodal fusion system includes the following steps: Step S1: Data Acquisition The smart insole device, which includes a flexible force sensor and an inertial measurement unit, collects the component data of the three-dimensional ground reaction force of the user's foot in real time through the three-dimensional component acquisition unit of the flexible force sensor, including vertical force Fz, anterior and posterior shear force Fx and lateral shear force Fy, and simultaneously collects foot kinematic data, including triaxial acceleration, angular velocity and attitude angle through the inertial measurement unit. Step S2: Communication Transmission The component data and kinematic data are synchronously transmitted to the monitoring terminal or data fusion module through the communication unit for subsequent time-series fusion and feature extraction. Step S3: Video Acquisition and Key Point Extraction A monocular camera captures continuous video images of the user during daily walking or standing, the video images covering the user's spine, hip, and shoulder areas; Based on the human walking spine posture recognition module, the video image is processed using an improved human key point detection model to obtain the coordinates of the user's human key points and extract posture parameters, including spinal curvature angle, pelvic rotation angle, and trunk tilt angle. Step S4: Multimodal data fusion The foot kinematics data and the user's key body data are spatiotemporally aligned and fused using the extended Kalman filter algorithm to obtain synchronously aligned multimodal temporal data. Step S5: Generate pseudo-labels and build dataset Based on the preset posture anomaly detection rules, multimodal time series data is analyzed. Abnormal moments in the multimodal time series data are used to generate pseudo-labels for training. A multimodal training dataset including component data, foot kinematic data, posture parameters and the pseudo-labels is established, that is, the posture parameters are converted into supervision signals that can be used for training. Step S6: Self-supervised pre-training and fine-tuning The scoliosis monitoring model is trained based on the multimodal training dataset. The scoliosis monitoring model adopts a strategy that combines self-supervised pre-training and supervised fine-tuning, and integrates temporal and spatial features. Step S7: Scoliosis Assessment and Output The newly collected multimodal time-series data is input into the trained scoliosis monitoring model, which outputs scoliosis assessment indicators, postural abnormality risk levels, and health recommendations to the user, thereby achieving intelligent monitoring and early warning of spinal health status.

[0011] Preferably, in step S1, the flexible force sensor uses a pressure sensing unit made of silicone and carbon nanowire composite material, which is arrayed in the forefoot and heel areas of the smart insole device.

[0012] Preferably, in step S3, the improved human key point detection model introduces optical flow constraints to improve the stability of human key points in consecutive frames and expand the key point detection nodes in the spinal region.

[0013] Preferably, in step S4, the extended Kalman filter algorithm uses the angular velocity in the foot kinematics data as the state input and the coordinates of the human body key points as the observation values. Through the iterative cycle of state prediction and observation update, the inertial measurement unit data and the video key point data are synchronized in time and space. After fusion, synchronized and aligned multimodal time-series data are obtained, which are used for subsequent posture analysis and training of the scoliosis monitoring model.

[0014] Preferably, in step S4, a GRF-attitude multimodal spatiotemporal fusion architecture is constructed, and the ground reaction force signal, inertial measurement signal, and attitude features are jointly input into the fusion network. The following fusion algorithm is used to achieve feature mapping and attitude estimation: in, This is a signal of ground reaction force. Characteristics of IMU inertial measurement; Pose angle features extracted for OpenPose; , , To integrate the weight matrix, an adaptive optimization algorithm is used for updating.

[0015] Preferably, in step S5, the abnormal posture detection rules include a spinal tilt angle greater than ±5°, a pelvic rotation angle greater than ±10°, and a left-right GRF asymmetry index of the two feet greater than 0.3.

[0016] Preferably, in step S6, the model employs a strategy combining self-supervised pre-training and supervised fine-tuning, using a contrastive learning loss function for feature alignment and model optimization. The contrastive learning loss function is defined as follows: in, This represents a similarity function between multimodal features. Temperature coefficient; We employ a self-supervised pre-training strategy to learn feature representations from unlabeled multimodal time-series data, and then perform supervised fine-tuning on labeled samples to optimize detection accuracy.

[0017] Preferably, in step S7, the scoliosis monitoring model adopts a hybrid deep neural network model, including a joint architecture of convolutional neural network and bidirectional long short-term memory network, for extracting spatial and temporal series features respectively.

[0018] Preferably, the abnormal posture detection rules include a spinal tilt angle greater than ±5°, a pelvic rotation angle greater than ±10°, and a left-right GRF asymmetry index of the two feet greater than 0.3.

[0019] Compared with the prior art, the present invention has the following beneficial effects: Reduce radiation risks and improve safety This invention uses non-invasive multimodal sensing to replace traditional imaging examinations such as X-rays, avoiding the health effects of ionizing radiation on patients, and is suitable for high-frequency daily posture monitoring.

[0020] Supports real-time monitoring in dynamic scenarios This invention achieves biomechanical assessment of the spine under dynamic motion by integrating three-dimensional ground reaction force (GRF) of the foot with trunk posture parameters, breaking through the blind spot of motion process data that cannot be obtained by traditional static images.

[0021] Achieving efficient fusion of multimodal sensing data This invention constructs a spatiotemporal fusion framework for GRF and attitude parameters, and uses extended Kalman filtering for timing alignment to improve the synchronization accuracy of multi-source data, thereby more accurately reflecting the correlation between spinal load transmission and posture abnormalities.

[0022] Significantly reduce annotation costs and manpower requirements This invention utilizes an improved OpenPose model combined with optical flow estimation and IMU co-localization to achieve automatic annotation of key pose features. Furthermore, it employs a self-supervised learning strategy for pre-training, effectively reducing reliance on manual labels and improving annotation efficiency by approximately 20 times.

[0023] Enhance the intelligent recognition capability of spinal posture abnormalities The hybrid deep neural network model trained by this invention has multimodal perception complementarity, which can accurately detect and classify the risk of posture abnormalities such as scoliosis and tilt in complex motion scenarios, with an accuracy comparable to that of traditional laboratory optical systems.

[0024] It has good wearability and practicality. This invention integrates a sensor module into a flexible smart insole, allowing users to collect data while walking or standing naturally, facilitating the practical promotion and application of health monitoring. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of a scoliosis monitoring method based on a multimodal fusion system, provided as an embodiment of the present invention.

[0026] The serial numbers in the diagram are as follows: 1. Three-dimensional component acquisition unit; 2. Inertial measurement unit; 3. Communication unit; 4. Monocular camera; 5. Human walking spine posture recognition module; 6. Scoliosis pre-training database set; 7. Scoliosis monitoring model. Detailed Implementation

[0027] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0028] like Figure 1 As shown in this embodiment, a method for monitoring scoliosis based on a multimodal fusion system is provided, which includes the following steps: Step S1: Data Acquisition (Acquisition of three-dimensional plantar component data and kinematic data) The smart insole device, which includes a flexible force sensor and an inertial measurement unit 2, collects the component data of the three-dimensional ground reaction force of the user's foot in real time through the three-dimensional component acquisition unit 1 of the flexible force sensor, including vertical force Fz, front and rear shear force Fx and lateral shear force Fy, and simultaneously collects foot kinematic data through the inertial measurement unit 2, including triaxial acceleration, angular velocity and attitude angle. The flexible force sensor uses a pressure sensing unit made of silicone and carbon nanowire composite material, which is arrayed in the forefoot and heel areas of the smart insole device.

[0029] The user wears a smart insole equipped with a flexible force sensor and an inertial measurement unit (2). The system collects plantar component data (GRF) in real time when the user is standing or walking, including vertical force (Fz), anterior and posterior shear force (Fx) and lateral shear force (Fy); and simultaneously collects foot kinematic data (IMU), including foot kinematic parameters such as triaxial acceleration, angular velocity and attitude angle.

[0030] Step S2: Communication Transmission The component data and kinematic data are synchronously transmitted to the monitoring terminal or data fusion module through the communication unit 3 for subsequent time-series fusion and feature extraction.

[0031] Step S3: Video Acquisition and Key Point Extraction A monocular camera 4 captures continuous video images of the user during daily walking or standing. The video images include multiple key skeletal points of the human body, especially the spine, hips, and shoulders, ensuring coverage of the main posture areas of the upper body.

[0032] Based on the human walking spine posture recognition module 5, the video image is processed using an improved human key point detection model to obtain the coordinates of the user's human key points and extract posture parameters, including spinal curvature angle, pelvic rotation angle, and trunk tilt angle.

[0033] Furthermore, in this embodiment, the improved human keypoint detection model introduces optical flow constraints to enhance the stability of human keypoints in consecutive frames and expand the keypoint detection nodes in the spinal region.

[0034] Step S4: Multimodal data fusion The foot kinematics data and the user's key body data are spatiotemporally aligned and fused using the extended Kalman filter algorithm to obtain synchronously aligned multimodal temporal data.

[0035] Because foot kinematics data (IMU) and camera data differ in sampling frequency, timestamps, and coordinate systems, the system uses the Extended Kalman Filter (EKF) algorithm to perform spatiotemporal alignment and data fusion, specifically including: (1) Time alignment: IMU high-frequency data is used as the main clock, and the timestamp of the camera image is interpolated and synchronized; (2) Spatial alignment: Establish a spatial transformation model between foot kinematics data (IMU) and key point coordinates, and unify it to the human body reference system; (3) Filtering and fusion: The EKF algorithm is used to eliminate noise interference during the motion process and output a stable fusion result.

[0036] The fused data constitutes a synchronized and registered multimodal time-series data sequence, covering plantar component data (GRF), foot kinematic data (IMU), and posture parameters at each moment.

[0037] The extended Kalman filter fusion algorithm uses the angular velocity of the foot kinematics data as the state input and the coordinates of the user's human body key points as the observation values ​​to achieve spatiotemporal synchronization between the inertial measurement unit and the video key point data.

[0038] Achieving multimodal sensing complementarity. The fusion of plantar component data (GRF) and IMU spatiotemporal data breaks through the one-sided perception of mechanical transition path or motion posture by a single sensor, and establishes a quantitative mapping relationship between plantar component data (GRF) and three-dimensional spinal deformities.

[0039] Step S5: Generate pseudo-labels and build dataset In the absence of manual annotation, the system analyzes multimodal time-series data based on preset posture anomaly judgment rules (such as spinal curvature angle exceeding a certain threshold, or continuous shift of pelvic rotation angle). Posture anomaly detection rules include spinal tilt angle greater than ±5°, pelvic rotation angle greater than ±10°, and left-right asymmetry index of bilateral foot plantar component data (GRF) greater than 0.3.

[0040] Based on the analysis of the fused multimodal time-series data, abnormal moments are automatically identified and pseudo-labels are generated for training. These pseudo-labels indicate potential postural anomalies in the data, providing supervision signals for subsequent deep learning models. Furthermore, the fused 3D ground reaction force plantar component data (GRF), foot kinematics data (IMU), posture parameters, and corresponding pseudo-labels are integrated to construct a multimodal training dataset for spinal anomaly detection. This training dataset features high temporal synchronization, strong spatial consistency, and low annotation cost, and is used to form a pre-training database set for scoliosis.

[0041] Step S6: Self-supervised pre-training and fine-tuning The scoliosis monitoring model (7) is trained based on the multimodal training dataset. The model adopts a strategy combining self-supervised pre-training and supervised fine-tuning to achieve joint learning of multimodal temporal and spatial features. The scoliosis monitoring model is a hybrid deep neural network structure, including a joint architecture of convolutional neural network (CNN) and bidirectional long short-term memory network (Bi-LSTM). The convolutional neural network is used to extract spatial features of the input data, and the bidirectional long short-term memory network is used to extract temporal features. The spatial and temporal features are fused to improve the accuracy of scoliosis detection.

[0042] The specific training method adopted is as follows: Self-supervised pre-training: using unlabeled data to perform feature comparison learning to obtain intermodal coupling representations; Supervised fine-tuning: Optimize detection performance based on a small amount of labeled data or high-confidence pseudo-labels; the final model can accurately identify abnormal posture patterns. Representation learning is performed on unlabeled multimodal temporal data (GRF, IMU); supervised fine-tuning optimizes the model's accuracy.

[0043] Step S7: Scoliosis Assessment and Output The newly collected multimodal time-series data is input into the trained scoliosis monitoring model 7, which outputs the user's scoliosis assessment indicators, postural abnormality risk level and health suggestions, so as to realize intelligent monitoring and early warning of spinal health status.

[0044] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0045] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0046] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for scoliosis monitoring based on a multi-modal fusion system, the method comprising: Comprising the following steps: Step S1: data acquisition Through the intelligent insole device containing a flexible force sensor and an inertial measurement unit (2), the three-dimensional component data of the user's foot bottom ground reaction force, including vertical force Fz, forward and backward shear force Fx, and lateral shear force Fy, are collected in real time through the three-dimensional component collection unit (1) of the flexible force sensor, and the foot kinematics data, including three-axis acceleration, angular velocity, and attitude angle, are collected synchronously through the inertial measurement unit (2); Step S2: communication transmission The component data and kinematics data are transmitted synchronously to the monitoring terminal or data fusion module through the communication unit (3) for subsequent time series fusion and feature extraction; Step S3: video acquisition and key point extraction Continuous video images of the user during daily walking or standing are collected through a monocular camera (4), and the video images cover the user's spine, hip, and shoulder regions; Based on the human walking spine posture recognition module (5), the improved human key point detection model is used to process the video images, obtain the coordinates of the user's human key points, and extract posture parameters, including spine bending angle, pelvic rotation angle, and trunk inclination angle; Step S4: multi-modal data fusion The foot kinematics data and the user's human key point data are time and space aligned and fused through the extended Kalman filter algorithm to obtain synchronized multi-modal time series data; Step S5: generating pseudo-labels and constructing data set Based on the preset posture anomaly detection rule, the multi-modal time series data is analyzed, the abnormal time in the multi-modal time series data generates pseudo-labels for training, and a multi-modal training data set of the scoliosis pre-training database set (6) including component data, foot kinematics data, posture parameters, and the pseudo-labels is established, i.e., the posture parameters are converted into supervised signals that can be used for training; Step S6: self-supervised pre-training and fine-tuning Based on the multi-modal training data set, the scoliosis monitoring model (7) is trained, which adopts a strategy combining self-supervised pre-training and supervised fine-tuning, and fuses time series and spatial features; Step S7: scoliosis evaluation and output The newly collected multi-modal time series data is input into the trained scoliosis monitoring model (7), and the user's scoliosis evaluation index, posture anomaly risk level, and health advice are output to realize intelligent monitoring and early warning of the spine health status.

2. The scoliosis monitoring method based on the multi-modal fusion system according to claim 1, characterized in that, In step S1, the flexible force sensor uses a pressure sensing unit made of silicone and carbon nanowire composite material, which is arrayed on the forefoot and heel of the intelligent insole device.

3. The scoliosis monitoring method based on the multi-modal fusion system according to claim 1, characterized in that, In step S3, the improved human key point detection model introduces optical flow constraints to improve the stability of human key points in consecutive frames and expand the key point detection nodes in the spine region.

4. The scoliosis monitoring method based on the multi-modal fusion system according to claim 1, characterized in that, In step S4, the extended Kalman filter algorithm uses angular velocity in foot kinematics data as state input and human key point coordinates as observation value, and through the cycle iteration of state prediction and observation update, the inertial measurement unit data and video key point data are synchronized in time and space; The fused multi-modal time-series data are synchronized and aligned, which are used for subsequent posture analysis and scoliosis monitoring model training.

5. The scoliosis monitoring method based on the multi-modal fusion system according to any one of claims 1 or 4, characterized in that, In step S4, a GRF-posture multi-modal spatio-temporal fusion architecture is constructed, the ground reaction force signal, the inertial measurement signal and the posture feature are jointly input into the fusion network, and the following fusion algorithm is used to realize feature mapping and posture estimation: wherein, is a ground reaction force signal; is an IMU inertial measurement feature; is a pose angle feature extracted by OpenPose; , , is a fusion weight matrix, which is updated by an adaptive optimization algorithm.

6. The scoliosis monitoring method based on the multi-modal fusion system according to claim 1, characterized in that, In step S5, the posture anomaly detection rules include that the scoliosis inclination angle is greater than ±5°, the pelvic rotation angle is greater than ±10°, and the bilateral GRF asymmetry index of the two feet is greater than 0.

3.

7. The scoliosis monitoring method based on the multi-modal fusion system according to claim 1, characterized in that, In step S6, the model adopts a combination strategy of self-supervised pre-training and supervised fine-tuning, and feature alignment and model optimization are performed through a contrastive learning loss function, which is defined as: wherein, represents a similarity function between the multi-modal features, is the temperature coefficient; Through the self-supervised pre-training strategy, feature representation learning of unlabeled multi-modal time-series data is realized, and supervised fine-tuning is performed on labeled samples to optimize the detection accuracy.

8. The scoliosis monitoring method based on the multi-modal fusion system according to claim 1, characterized in that, In step S7, the scoliosis monitoring model (7) adopts a hybrid deep neural network model, including a convolutional neural network and a bidirectional long short-term memory network joint architecture, which is used to extract spatial and temporal sequence features respectively.

9. The scoliosis monitoring method based on the multi-modal fusion system according to claim 1, characterized in that, The posture anomaly detection rules include that the scoliosis inclination angle is greater than ±5°, the pelvic rotation angle is greater than ±10°, and the bilateral GRF asymmetry index of the two feet is greater than 0.3.

Images