Scoliosis assessment method and system based on multi-modal data

Abstract

The invention discloses a scoliosis assessment method and system based on multi-modal data. The method comprises the steps that plantar pressure static features and plantar pressure dynamic features are extracted according to plantar pressure signals of a to-be-assessed target; extracting surface electromyogram static features and surface electromyogram dynamic features according to the obtained surface electromyogram signals; ultrasonic image static features and ultrasonic image dynamic features are extracted according to the obtained ultrasonic image; inputting the plantar pressure static feature, the plantar pressure dynamic feature, the surface myoelectricity static feature, the surface myoelectricity dynamic feature, the ultrasonic image static feature and the ultrasonic image dynamic feature into a preset scoliosis feature fusion model for fusion to obtain a static fusion feature and a dynamic fusion feature; and determining the scoliosis grade of the to-be-evaluated target according to a comprehensive evaluation score and a preset scoliosis grade table, wherein the comprehensive evaluation score is obtained by calculating the static fusion features, the dynamic fusion features and a preset weight coefficient. According to the invention, the scoliosis evaluation accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of scoliosis detection, and in particular to a scoliosis assessment method and system based on multimodal data. Background Art

[0002] Scoliosis is a complex disease characterized by three-dimensional deformity of the spine. Its early and accurate assessment is of great significance for delaying the progression of the disease and formulating individualized treatment plans. At present, traditional methods mainly rely on a single biomechanical detection technology, such as plantar pressure measurement or surface electromyography. Plantar pressure measurement mainly reflects the mechanical distribution of the lower limbs when they are loaded, while surface electromyography focuses on the electrophysiological signals of muscle activity. The two are independent of each other and lack a unified framework for integrated analysis. With the increasing clinical needs and the continuous advancement of medical technology, single-modality detection methods have gradually revealed their limitations. This single-modality detection and evaluation lacks multimodal fusion analysis, which leads to a one-sided comprehensive assessment of scoliosis and makes it difficult to achieve the accuracy of scoliosis assessment. Summary of the invention

[0003] The embodiments of the present invention provide a scoliosis assessment method and system based on multimodal data, which can effectively solve the problem that the prior art uses single-modal detection and assessment and lacks multimodal fusion analysis, resulting in a one-sided comprehensive assessment of scoliosis and difficulty in achieving the accuracy of scoliosis assessment.

[0004] An embodiment of the present invention provides a scoliosis assessment method based on multimodal data, comprising: Acquire plantar pressure signals, surface electromyography signals and ultrasound images of the target to be evaluated; Extract the pressure center symmetry coefficient, the front and rear foot weight ratio and the lateral balance coefficient according to the plantar pressure signal to obtain the static characteristics of the plantar pressure; extract the pressure center movement trajectory, gait cycle parameters and joint movement trajectory to obtain the dynamic characteristics of the plantar pressure; Extract the electromyographic median frequency, electromyographic average power frequency, electromyographic average amplitude and electromyographic spectrum area according to the surface electromyographic signal to obtain the surface electromyographic static characteristics; extract the integral electromyographic value change rate and the electromyographic signal change period to obtain the surface electromyographic dynamic characteristics; The bilateral paravertebral skin thickness, fascia thickness, erector spinae muscle cross-sectional area, erector spinae muscle circumference and muscle tissue elastic coefficient are extracted from the ultrasound image to obtain the static features of the ultrasound image; the tissue strain rate and tissue elastic modulus are extracted to obtain the dynamic features of the ultrasound image; Inputting the static features of plantar pressure, the dynamic features of plantar pressure, the static features of surface electromyography, the dynamic features of surface electromyography, the static features of ultrasound images, and the dynamic features of ultrasound images into a preset scoliosis feature fusion model for feature fusion to obtain static fusion features and dynamic fusion features; Calculate according to the static fusion feature, the dynamic fusion feature and a preset weight coefficient to obtain a comprehensive evaluation score; The scoliosis grade of the target to be evaluated is determined based on the comprehensive evaluation score and a preset scoliosis grade table.

[0005] Furthermore, the training of the scoliosis feature fusion model includes: Acquire historical plantar pressure signals, historical surface electromyography signals, historical ultrasound images, historical static fusion features, historical dynamic fusion features and initial model parameters of the scoliosis target; Extract the pressure center symmetry coefficient, the front and rear foot weight ratio and the lateral balance coefficient according to the historical plantar pressure signal to obtain the historical plantar pressure static characteristics; extract the pressure center movement trajectory, gait cycle parameters and joint movement trajectory to obtain the historical plantar pressure dynamic characteristics; Extract the median EMG frequency, the average EMG power frequency, the average EMG amplitude and the EMG spectrum area according to the historical surface EMG signal to obtain the historical surface EMG static characteristics; extract the integral EMG value change rate and the EMG signal change period to obtain the historical surface EMG dynamic characteristics; According to the historical ultrasound images, the bilateral paravertebral skin thickness, fascia thickness, erector spinae muscle cross-sectional area, erector spinae muscle circumference and muscle tissue elastic coefficient are extracted to obtain the static features of the historical ultrasound images; the tissue strain rate and tissue elastic modulus are extracted to obtain the dynamic features of the historical ultrasound images; Input the historical plantar pressure static features, historical plantar pressure dynamic features, historical surface electromyography static features, historical surface electromyography dynamic features, historical ultrasound image static features, and historical ultrasound image dynamic features into the scoliosis feature fusion model to be trained, perform iterative training, and obtain predicted static fusion features and predicted dynamic fusion features; Calculating a loss function value of a preset model loss function according to the predicted static fusion feature, the predicted dynamic fusion feature, the historical static fusion feature, and the historical dynamic fusion feature; When the loss function value is determined to be convergent, a trained scoliosis feature fusion model is obtained; When it is determined that the loss function value has not converged, the current model parameters are updated according to the model loss function; and the updated current model parameters are used as the current model parameters for the next training; Among them, the current model parameters at the initial time are the initial model parameters.

[0006] Furthermore, according to the plantar pressure signal, the pressure center symmetry coefficient, the front and rear foot weight ratio and the lateral balance coefficient are extracted to obtain the static characteristics of the plantar pressure; the pressure center movement trajectory, gait cycle parameters and joint movement trajectory are extracted to obtain the dynamic characteristics of the plantar, including: Determining the left foot load pressure point distribution and the right foot load pressure point distribution according to the plantar pressure signal; The pressure center symmetry coefficient is calculated according to the left foot load pressure point distribution and the right foot load pressure point distribution; the forefoot pressure sum and the rear foot pressure sum are calculated according to the left foot load pressure point distribution and the right foot load pressure point distribution respectively; and the forefoot and rearfoot weight ratio is calculated according to the forefoot pressure sum and the rear foot pressure sum; the plantar lateral pressure sum and the plantar medial pressure sum are calculated according to the left foot load pressure point distribution and the right foot load pressure point distribution respectively; and the plantar lateral balance coefficient is calculated according to the plantar lateral pressure sum and the plantar medial pressure sum; The pressure center symmetry coefficient, the front and rear foot weight ratio and the lateral balance coefficient are used as static characteristics of plantar pressure; Determine the coordinate change of the pressure center according to the plantar pressure signal to obtain the movement trajectory of the pressure center; Determining the starting point and the ending point of the gait cycle according to the plantar pressure signal; Calculate the gait cycle duration, the stance phase duration, and the swing phase duration according to the starting point and the end point to obtain gait cycle parameters; Extracting joint motion trajectory according to the plantar pressure signal; The pressure center movement trajectory, the gait cycle parameters and the joint movement trajectory are used as dynamic features of plantar pressure.

[0007] Further, according to the surface electromyography signal, the electromyography median frequency, the electromyography average power frequency, the electromyography average amplitude and the electromyography spectrum area are extracted to obtain the surface electromyography static characteristics; the integral electromyography value change rate and the electromyography signal change period are extracted to obtain the surface electromyography dynamic characteristics, including: Performing Fourier transform on the surface electromyography signal to obtain an electromyography spectrum diagram; According to the electromyographic spectrum, determining the electromyographic median frequency, the electromyographic average power frequency, the electromyographic average amplitude and the electromyographic spectrum area; The EMG median frequency, the EMG average power frequency, the EMG average amplitude and the EMG spectrum area are used as surface EMG static features; Performing segmented processing according to the surface electromyographic signal and a preset time window to obtain a plurality of segmented electromyographic signals; According to the segmented electromyographic signals, the integrated electromyographic value of each segmented electromyographic signal is calculated; and according to the integrated electromyographic value, a calculation is performed to obtain the integrated electromyographic value change rate; Performing time-frequency analysis on the surface electromyographic signal to extract the electromyographic signal variation period; The integrated electromyographic value change rate and the electromyographic signal change period are used as surface electromyographic dynamic characteristics.

[0008] Furthermore, the bilateral paravertebral skin thickness, fascia thickness, erector spinae muscle cross-sectional area, erector spinae muscle circumference and muscle tissue elastic coefficient are extracted from the ultrasound image to obtain the static features of the ultrasound image; the tissue strain rate and tissue elastic modulus are extracted to obtain the dynamic features of the ultrasound image, including: Determine the skin layer, fascia layer and erector spinae muscle contour according to the ultrasound image; Determine the thickness of the skin layer according to the skin layer; and calculate the bilateral paravertebral skin thickness according to the thickness of the skin layer; determining a fascia thickness according to the fascia layer; According to the erector spinae muscle contour, the cross-sectional area of ​​the erector spinae muscle and the circumference of the erector spinae muscle are calculated; Determine the elasticity distribution map of the muscle tissue according to the ultrasonic image; and calculate the elasticity coefficient of the muscle tissue according to the elasticity distribution map; The bilateral paravertebral skin thickness, fascia thickness, erector spinae muscle cross-sectional area, erector spinae muscle circumference and muscle tissue elasticity coefficient are used as static features of ultrasound images; Determine tissue movement speed and tissue deformation parameters based on the ultrasound image Calculating the tissue strain rate according to the tissue movement speed; Calculating the tissue elastic modulus according to the tissue deformation parameter; The tissue strain rate and the tissue elastic modulus are used as dynamic features of ultrasound images.

[0009] Furthermore, the weight coefficient includes a static feature weight coefficient and a dynamic feature weight coefficient; The static fusion feature, the dynamic fusion feature and the preset weight coefficient are used to calculate and obtain a comprehensive evaluation score, including: Performing normalization processing according to the static fusion feature and the dynamic fusion feature to obtain a target static fusion feature and a target dynamic fusion feature; The static feature evaluation score is calculated by multiplying the target static fusion feature and the static feature weight coefficient; the dynamic feature evaluation score is calculated by multiplying the target dynamic fusion feature and the dynamic feature weight coefficient; the comprehensive evaluation score is calculated by adding the static feature evaluation score and the dynamic feature evaluation score.

[0010] Further, the scoliosis grade table includes: assessment score intervals and scoliosis grades; Determining the scoliosis grade of the target to be evaluated according to the comprehensive evaluation score and a preset scoliosis grade table, including: traversing the evaluation score interval in the preset scoliosis grade table according to the comprehensive evaluation score to determine the corresponding evaluation score interval; According to the corresponding assessment score range, the corresponding scoliosis grade is determined.

[0011] Furthermore, it also includes: Determine corresponding treatment measures according to the scoliosis grade; An assessment report is generated based on the comprehensive assessment score, the scoliosis grade and the corresponding treatment measures, so that corresponding treatment measures are taken for the assessment target according to the assessment report.

[0012] Furthermore, it also includes: The plantar pressure signal is collected by a pressure sensor array; The surface electromyography signal is collected by wireless multi-lead surface electrodes; The ultrasonic image is collected by a high-frequency ultrasonic probe.

[0013] As an improvement of the above solution, another embodiment of the present invention provides a scoliosis assessment system based on multimodal data, including: A data acquisition module, used to acquire the plantar pressure signal, surface electromyography signal and ultrasonic image of the target to be evaluated; The first feature extraction module is used to extract the pressure center symmetry coefficient, the front and rear foot weight ratio and the lateral balance coefficient according to the plantar pressure signal to obtain the static characteristics of the plantar pressure; extract the pressure center movement trajectory, gait cycle parameters and joint movement trajectory to obtain the dynamic characteristics of the plantar pressure; The second feature extraction module is used to extract the electromyographic median frequency, the electromyographic average power frequency, the electromyographic average amplitude and the electromyographic spectrum area according to the surface electromyographic signal to obtain the surface electromyographic static features; extract the integral electromyographic value change rate and the electromyographic signal change period to obtain the surface electromyographic dynamic features; The third feature extraction module is used to extract the bilateral paravertebral skin thickness, fascia thickness, erector spinae muscle cross-sectional area, erector spinae muscle circumference and muscle tissue elasticity coefficient according to the ultrasound image to obtain the static features of the ultrasound image; extract the tissue strain rate and tissue elastic modulus to obtain the dynamic features of the ultrasound image; A feature fusion module, for inputting the static feature of plantar pressure, the dynamic feature of plantar pressure, the static feature of surface electromyography, the dynamic feature of surface electromyography, the static feature of ultrasound image and the dynamic feature of ultrasound image into a preset scoliosis feature fusion model for feature fusion, so as to obtain static fusion features and dynamic fusion features; A comprehensive evaluation module, used to calculate according to the static fusion feature, the dynamic fusion feature and a preset weight coefficient to obtain a comprehensive evaluation score; The scoliosis grade determination module is used to determine the scoliosis grade of the target to be evaluated based on the comprehensive evaluation score and a preset scoliosis grade table.

[0014] Another embodiment of the present invention provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, a scoliosis assessment method based on multimodal data as described in the above embodiment is implemented.

[0015] Another embodiment of the present invention provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a scoliosis assessment method based on multimodal data as described in the above embodiment.

[0016] By implementing the present invention, at least the following beneficial effects are achieved: The present invention provides a scoliosis assessment method and system based on multimodal data. The method can obtain a plantar pressure signal, a surface electromyography signal and an ultrasonic image of a target to be assessed; extract the pressure center symmetry coefficient, the front and rear foot weight ratio and the lateral balance coefficient according to the plantar pressure signal to obtain the plantar pressure static characteristics; extract the pressure center movement trajectory, gait cycle parameters and joint movement trajectory to obtain the plantar pressure dynamic characteristics; extract the electromyography median frequency, the electromyography average power frequency, the electromyography average amplitude and the electromyography spectrum area according to the surface electromyography signal to obtain the surface electromyography static characteristics; extract the integral electromyography value change rate and the electromyography signal change cycle to obtain the surface electromyography dynamic characteristics; extract the bilateral paravertebral skin thickness according to the ultrasonic image. The static features of ultrasound images are obtained by extracting the plantar pressure, fascia thickness, erector spinae cross-sectional area, erector spinae circumference and elastic coefficient of muscle tissue; the tissue strain rate and tissue elastic modulus are extracted to obtain the dynamic features of ultrasound images; the static features of plantar pressure, the dynamic features of plantar pressure, the static features of surface electromyography, the dynamic features of surface electromyography, the static features of ultrasound images and the dynamic features of ultrasound images are input into the preset scoliosis feature fusion model for feature fusion to obtain static fusion features and dynamic fusion features; the static fusion features, the dynamic fusion features and the preset weight coefficients are calculated to obtain a comprehensive evaluation score; the scoliosis grade of the target to be evaluated is determined according to the comprehensive evaluation score and the preset scoliosis grade table. The data from three different sources, plantar pressure signals, surface electromyography signals and ultrasound images, reflect different aspects of human body movement, muscle activity and internal structure. The features from different sources are fused to obtain static fusion features and dynamic fusion features, which can more comprehensively evaluate the health of the spine, thereby improving the accuracy of scoliosis evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of a scoliosis assessment method based on multimodal data provided by one embodiment of the present invention; Figure 2 is a flowchart of scoliosis assessment and treatment provided by an embodiment of the present invention; Figure 3 It is a structural schematic diagram of a scoliosis assessment system based on multimodal data provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] See also Figure 1 , is a flowchart of a scoliosis assessment method based on multimodal data provided by an embodiment of the present invention, comprising: S1, obtaining the plantar pressure signal, surface electromyography signal and ultrasound image of the target to be evaluated; Specifically, the plantar pressure signal is collected by a pressure sensor array; the surface electromyography signal is collected by a wireless multi-lead surface electrode; and the ultrasound image is collected by a high-frequency ultrasound probe.

[0020] In a preferred embodiment of the present invention, the pressure sensor array can arrange a large number of sensor points in the effective area of ​​the sole of the target foot to be evaluated, and provide real-time feedback on the load pressure, posture information and body balance of the foot, and provide accurate plantar pressure distribution data; it can cover multiple areas of the sole, including the forefoot, midfoot and hindfoot, so as to more comprehensively evaluate the changes in plantar pressure. Wireless multi-lead surface electrodes do not need to pierce the skin when collecting electromyographic signals, reducing harm and discomfort to patients; surface electrodes can collect electrical signals generated by muscle activity in real time to reflect the dynamic changes of muscles; multi-lead design can collect electromyographic signals of multiple muscles at the same time, providing richer muscle activity information. High-frequency ultrasonic probes can provide high-resolution images, clearly showing the structure of the spine and its surrounding tissues; ultrasonic examination is a non-invasive examination method that will not cause radiation damage to patients and is highly safe.

[0021] S2, extracting the pressure center symmetry coefficient, the front and rear foot weight ratio and the lateral balance coefficient according to the plantar pressure signal to obtain the static characteristics of the plantar pressure; extracting the pressure center movement trajectory, gait cycle parameters and joint movement trajectory to obtain the dynamic characteristics of the plantar pressure; Preferably, the sampling frequency of the plantar pressure signal is 100 Hz, and then the zero drift is corrected, the 50 Hz power frequency interference is filtered out, and the pressure threshold is set (greater than 0.1 N / cm 2 ), spatial interpolation processing (bilinear interpolation algorithm) preprocessing steps to obtain the final plantar pressure signal. The surface electromyography signal sampling frequency is 100Hz, and then it is preprocessed by bandpass filtering (20-500Hz), power frequency notch (50Hz), baseline drift correction, RMS (root mean square) calculation (time window: 100ms) to obtain the final surface electromyography signal. The image resolution of the ultrasound image is greater than or equal to 1024×768 pixels. After image denoising (Gaussian filtering), contrast enhancement, edge enhancement, and image registration preprocessing steps, the final ultrasound image is obtained.

[0022] Specifically, according to the plantar pressure signal, the pressure center symmetry coefficient, the front and rear foot weight ratio and the lateral balance coefficient are extracted to obtain the static characteristics of the plantar pressure; the pressure center movement trajectory, gait cycle parameters and joint movement trajectory are extracted to obtain the dynamic characteristics of the plantar, including: according to the plantar pressure signal, the left foot load pressure point distribution and the right foot load pressure point distribution are determined; according to the left foot load pressure point distribution and the right foot load pressure point distribution, the pressure center symmetry coefficient is calculated; according to the left foot load pressure point distribution and the right foot load pressure point distribution, the forefoot pressure sum and the rear foot pressure sum are calculated respectively; and the front and rear foot weight ratio is calculated according to the forefoot pressure sum and the rear foot pressure sum; according to the left foot load pressure point distribution and the right foot load pressure point distribution, the left foot load pressure point distribution and the right foot load pressure point distribution are calculated; , respectively calculate the sum of the lateral pressure of the plantar and the sum of the medial pressure of the plantar; and calculate according to the sum of the lateral pressure of the plantar and the sum of the medial pressure of the plantar to obtain the lateral balance coefficient; use the pressure center symmetry coefficient, the front and rear foot load ratio and the lateral balance coefficient as static characteristics of plantar pressure; determine the coordinate change of the pressure center according to the plantar pressure signal to obtain the pressure center movement trajectory; determine the starting point and the end point of the gait cycle according to the plantar pressure signal; calculate the gait cycle duration, the support phase duration and the swing phase duration according to the starting point and the end point to obtain the gait cycle parameters; extract the joint movement trajectory according to the plantar pressure signal; use the pressure center movement trajectory, the gait cycle parameters and the joint movement trajectory as dynamic characteristics of plantar pressure.

[0023] In a preferred embodiment of the present invention, after the plantar pressure signal is standardized and noise-reduced, the load pressure point distribution of the left foot and the load pressure point distribution of the right foot are determined, and then the pressure center symmetry coefficient, the front and rear foot weight ratio and the lateral balance coefficient are calculated respectively according to the left foot load pressure point distribution and the right foot load pressure point distribution. The pressure center symmetry coefficient reflects the symmetry of the left and right feet in the load pressure distribution. A higher symmetry coefficient usually means that the pressure distribution of the left and right feet of the target to be evaluated is more balanced when walking or standing. The front and rear foot weight ratio can understand the force of the front and rear feet of the target to be evaluated when walking or standing. The lateral balance coefficient reflects the stability of the foot in the lateral direction. Then the pressure center symmetry coefficient, the front and rear foot weight ratio and the lateral balance coefficient are used as the static characteristics of plantar pressure. Then, according to the plantar pressure signal, the pressure center movement trajectory, gait cycle parameters and joint movement trajectory are extracted; and the pressure center movement trajectory, the gait cycle parameters and the joint movement trajectory are used as the dynamic characteristics of plantar pressure. The pressure center movement trajectory can intuitively display the dynamic changes of the plantar pressure center of an individual during walking. Combining the static and dynamic characteristics of plantar pressure can provide a more comprehensive understanding of the gait characteristics and foot health of the target to be evaluated. There is a close relationship between plantar pressure characteristics and scoliosis. By analyzing plantar pressure characteristics, it can provide important clues and basis for the evaluation of scoliosis.

[0024] S3, extracting the electromyographic median frequency, the electromyographic average power frequency, the electromyographic average amplitude and the electromyographic spectrum area according to the surface electromyographic signal to obtain the surface electromyographic static characteristics; extracting the integral electromyographic value change rate and the electromyographic signal change period to obtain the surface electromyographic dynamic characteristics; Specifically, according to the surface electromyographic signal, the electromyographic median frequency, the electromyographic average power frequency, the electromyographic average amplitude and the electromyographic spectrum area are extracted to obtain the surface electromyographic static characteristics; the integral electromyographic value change rate and the electromyographic signal change period are extracted to obtain the surface electromyographic dynamic characteristics, including: performing Fourier transform according to the surface electromyographic signal to obtain an electromyographic spectrum diagram; determining the electromyographic median frequency, the electromyographic average power frequency, the electromyographic average amplitude and the electromyographic spectrum area according to the electromyographic spectrum diagram; using the electromyographic median frequency, the electromyographic average power frequency, the electromyographic average amplitude and the electromyographic spectrum area as the surface electromyographic static characteristics; performing segmentation processing according to the surface electromyographic signal and a preset time window to obtain a plurality of segmented electromyographic signals; calculating the integral electromyographic value of each segmented electromyographic signal according to the segmented electromyographic signal; and calculating according to the integral electromyographic value to obtain the integral electromyographic value change rate; performing time-frequency analysis according to the surface electromyographic signal to extract the electromyographic signal change period; using the integral electromyographic value change rate and the electromyographic signal change period as the surface electromyographic dynamic characteristics.

[0025] In a preferred embodiment of the present invention, the EMG median frequency (MF) and the EMG mean power frequency (MPF) are important indicators reflecting the fatigue degree of the paraspinal muscles. By extracting these two features, the fatigue state of the paraspinal muscles can be accurately assessed, providing basic data for scoliosis assessment. The average EMG amplitude reflects the intensity of muscle activity, which helps to understand the contraction of the muscle in a static state. The EMG spectrum area provides comprehensive information on the frequency distribution of muscle activity, which helps to analyze the complexity of muscle activity. The integrated EMG value change rate reflects the rate of change of the activity intensity of the muscle within a certain period of time, and is an important indicator for evaluating muscle endurance. The EMG signal change cycle reveals the periodic law of muscle activity, which helps to understand the rhythm and pattern of muscle activity. Combining the surface EMG static features and the surface EMG dynamic features can more comprehensively reflect the functional state of the muscle. The surface EMG static features mainly focus on the activity of the muscle in a specific state, while the surface EMG dynamic features reveal the changing laws and trends of the muscle during the activity process. There is a close relationship between surface electromyography characteristics and scoliosis. By analyzing the surface electromyography characteristics, we can understand the activity of the muscles around the spine and provide a basis for muscle activity for the assessment of scoliosis. For example, muscle imbalance or fatigue may lead to the aggravation or deterioration of scoliosis.

[0026] S4, extracting bilateral paravertebral skin thickness, fascia thickness, erector spinae muscle cross-sectional area, erector spinae muscle circumference and muscle tissue elasticity coefficient according to the ultrasound image to obtain the static features of the ultrasound image; extracting tissue strain rate and tissue elastic modulus to obtain the dynamic features of the ultrasound image; Preferably, bilateral paravertebral skin thickness, fascia thickness, erector spinae cross-sectional area, erector spinae circumference and muscle tissue elastic coefficient are extracted from the ultrasound image to obtain the static features of the ultrasound image; tissue strain rate and tissue elastic modulus are extracted to obtain the dynamic features of the ultrasound image, including: determining the skin layer, fascia layer and erector spinae muscle contour according to the ultrasound image; determining the skin layer thickness according to the skin layer; and calculating the bilateral paravertebral skin thickness according to the skin layer thickness; determining the fascia thickness according to the fascia layer; and calculating the erector spinae muscle cross-sectional area and erector spinae muscle circumference according to the erector spinae muscle contour; According to the ultrasound image, the elasticity distribution map of the muscle tissue is determined; and the elasticity coefficient of the muscle tissue is calculated according to the elasticity distribution map; the bilateral paravertebral skin thickness, fascia thickness, erector spinae muscle cross-sectional area, erector spinae muscle circumference and muscle tissue elasticity coefficient are used as static features of the ultrasound image; according to the ultrasound image, the tissue movement speed and tissue deformation parameters are determined; according to the tissue movement speed, the tissue strain rate is calculated; according to the tissue deformation parameters, the tissue elastic modulus is calculated; the tissue strain rate and the tissue elastic modulus are used as dynamic features of the ultrasound image.

[0027] In a preferred embodiment of the present invention, the bilateral paravertebral skin thickness and fascia thickness can reflect the health of the soft tissue around the spine, which is of great significance for evaluating the stability of the spine. The erector spinae cross-sectional area and the erector spinae circumference can quantify the size and shape of the erector spinae (i.e., the muscles on both sides of the spine), which helps to understand the strength and endurance of the muscles. The elastic coefficient of muscle tissue can reflect the elastic state of the muscle and is an important indicator for evaluating the muscle's recovery ability and fatigue resistance. The strain rate obtained by calculating the tissue movement speed can reflect the deformation speed and recovery ability of the muscle and soft tissue, which helps to evaluate its dynamic performance. The elastic modulus calculated based on the tissue deformation parameters can quantify the stiffness and elasticity of the tissue, which is of great significance for evaluating the mechanical properties and health of the tissue. By extracting ultrasonic image features, an objective and quantitative basis can be provided for the diagnosis of diseases such as scoliosis.

[0028] S5, inputting the static feature of plantar pressure, the dynamic feature of plantar pressure, the static feature of surface electromyography, the dynamic feature of surface electromyography, the static feature of ultrasound image and the dynamic feature of ultrasound image into a preset scoliosis feature fusion model for feature fusion, and obtaining static fusion features and dynamic fusion features; Specifically, the training of the scoliosis feature fusion model includes: obtaining the historical plantar pressure signal, historical surface electromyography signal, historical ultrasound image, historical static fusion feature, historical dynamic fusion feature and initial model parameters of the scoliosis target; extracting the pressure center symmetry coefficient, the front and rear foot weight ratio and the lateral balance coefficient according to the historical plantar pressure signal to obtain the historical plantar pressure static feature; extracting the pressure center movement trajectory, gait cycle parameters and joint movement trajectory to obtain the historical plantar pressure dynamic feature; extracting the electromyography median frequency, electromyography average power frequency, electromyography average amplitude and electromyography spectrum area according to the historical surface electromyography signal to obtain the historical surface electromyography static feature; extracting the integrated electromyography value change rate and the electromyography signal change cycle to obtain the historical surface electromyography dynamic feature; extracting the bilateral paravertebral skin thickness, fascia thickness, erector spinae muscle cross-sectional area, erector spinae muscle circumference and muscle tissue elasticity coefficient according to the historical ultrasound image to obtain the historical ultrasound image static feature ; Extract tissue strain rate and tissue elastic modulus to obtain dynamic features of historical ultrasound images; input the historical static features of plantar pressure, the historical dynamic features of plantar pressure, the historical static features of surface electromyography, the historical dynamic features of surface electromyography, the historical static features of ultrasound images, and the historical dynamic features of ultrasound images into the scoliosis feature fusion model to be trained, perform iterative training, and obtain predicted static fusion features and predicted dynamic fusion features; calculate the loss function value of the preset model loss function according to the predicted static fusion features, the predicted dynamic fusion features, the historical static fusion features, and the historical dynamic fusion features; when it is determined that the loss function value converges, obtain the trained scoliosis feature fusion model; when it is determined that the loss function value does not converge, update the current model parameters according to the model loss function; and use the updated current model parameters as the current model parameters for the next training; wherein, the current model parameters at the initial time are the initial model parameters.

[0029] In a preferred embodiment of the present invention, through an iterative training process, the model can gradually adjust itself according to historical data and current parameters to minimize the preset model loss function value. According to the historical plantar pressure static features, historical plantar pressure dynamic features, historical surface electromyography static features, historical surface electromyography dynamic features, historical ultrasound image static features and historical ultrasound image dynamic features, and the current model parameters, iterative training is performed to obtain predicted static fusion features and predicted dynamic fusion features, and according to the loss function value of the preset model loss function, the current model parameters of the scoliosis feature fusion model are continuously adjusted until the loss function value converges. For example, a machine learning algorithm (such as support vector machine, random forest, etc.) can be used to train the scoliosis feature fusion model, thereby realizing multimodal feature fusion.

[0030] In a preferred embodiment of the present invention, the static characteristics of plantar pressure (pressure center symmetry coefficient, front and rear foot weight ratio, lateral balance coefficient), surface electromyography static characteristics (EMG median frequency, EMG average power frequency, EMG average amplitude, EMG spectrum area) and ultrasound image static characteristics (bilateral paravertebral skin thickness, fascia thickness, erector spinae cross-sectional area, erector spinae circumference, muscle tissue elasticity coefficient) are standardized to eliminate dimensional differences. Then, principal component analysis (PCA) or linear discriminant analysis (LDA) is used to reduce the dimension of the standardized static features, reduce the feature dimension and retain the main information, and then the static features of different data sources are weighted and summed to generate static fusion features. The formula is: Among them, w i is the weight coefficient of the i-th static feature, F i is the i-th standardized static feature. The weighted fused static fusion features are expressed as vectors. The dynamic features of plantar pressure (pressure center movement trajectory, gait cycle parameters, joint movement trajectory), surface electromyography dynamic features (integrated electromyography value change rate, electromyography signal change cycle) and ultrasound image dynamic features (tissue strain rate, tissue elastic modulus) are standardized to eliminate dimensional differences. If the time series lengths of the dynamic features are inconsistent, use interpolation or truncation methods to align the time series to ensure consistent feature dimensions. The dynamic features of different data sources are weighted and summed to generate dynamic fusion features. The formula is: Among them, w i is the weight coefficient of the i-th feature, D i is the i-th normalized dynamic feature. The dynamic fusion feature after weighted fusion is expressed as a vector.

[0031] S6. Calculate according to the static fusion feature, the dynamic fusion feature and a preset weight coefficient to obtain a comprehensive evaluation score; Specifically, the weight coefficient includes a static feature weight coefficient and a dynamic feature weight coefficient; a comprehensive evaluation score is obtained by calculation based on the static fusion feature, the dynamic fusion feature and the preset weight coefficient, including: normalizing the static fusion feature and the dynamic fusion feature to obtain the target static fusion feature and the target dynamic fusion feature; multiplying the target static fusion feature and the static feature weight coefficient to calculate the static feature evaluation score; multiplying the target dynamic fusion feature and the dynamic feature weight coefficient to calculate the dynamic feature evaluation score; and adding the static feature evaluation score and the dynamic feature evaluation score to calculate the comprehensive evaluation score.

[0032] In a preferred embodiment of the present invention, the static feature weight coefficient is set to 40% and the dynamic feature weight coefficient is set to 60%. The static fusion feature and the dynamic fusion feature are first normalized to unify the dimension and the numerical range to obtain the target static fusion feature and the target dynamic fusion feature. Then, the static feature evaluation score is calculated by multiplying the target static fusion feature and the static feature weight coefficient; the dynamic feature evaluation score is calculated by multiplying the target dynamic fusion feature and the dynamic feature weight coefficient; and finally, the comprehensive evaluation score is calculated by adding the static feature evaluation score and the dynamic feature evaluation score. For example, the static feature evaluation score = Σ (target static fusion feature × static feature weight coefficient); the dynamic feature evaluation score = Σ (target dynamic fusion feature × dynamic feature weight coefficient); the comprehensive evaluation score = static feature evaluation score × 0.4 + dynamic feature evaluation score × 0.6. The introduction of the static feature weight coefficient and the dynamic feature weight coefficient can be weighted according to the importance of different features in the evaluation, so that the evaluation result is more in line with the actual situation. This processing method helps to avoid the excessive influence of a single feature on the evaluation result and improve the accuracy and reliability of the evaluation. Since the static feature weight coefficient and the dynamic feature weight coefficient can be adjusted according to the actual situation, personalized analysis can be performed according to the characteristics of different targets to be evaluated, which helps to more accurately understand the actual situation of the target to be evaluated.

[0033] S7. Determine the scoliosis grade of the target to be evaluated based on the comprehensive evaluation score and a preset scoliosis grade table.

[0034] Specifically, the scoliosis grade table includes: an assessment score interval and a scoliosis grade; according to the comprehensive assessment score and a preset scoliosis grade table, the scoliosis grade of the target to be assessed is determined, including: according to the comprehensive assessment score, traversing the assessment score interval in the preset scoliosis grade table to determine the corresponding assessment score interval; according to the corresponding assessment score interval, determining the corresponding scoliosis grade.

[0035] In a preferred embodiment of the present invention, the assessment score intervals are divided into: (full score 100 points): Excellent: 85-100 points; Good: 70-84 points; Average: 55-69 points; Poor: 40-54 points; Severe: <40 points. The assessor only needs to find the corresponding assessment score interval in the grade table according to the comprehensive assessment score to quickly determine the scoliosis grade, which simplifies the assessment process and improves the assessment efficiency. The scoliosis grade table contains multiple assessment score intervals and corresponding scoliosis grades, which helps to more accurately assess the degree of scoliosis and provide a basis for formulating targeted treatment plans.

[0036] Schematically, it also includes: determining corresponding treatment measures according to the scoliosis grade; generating an assessment report according to the comprehensive assessment score, the scoliosis grade and the corresponding treatment measures, so that corresponding treatment measures are taken for the assessment target according to the assessment report.

[0037] In a preferred embodiment of the present invention, 85-100 points: observation and follow-up, maintaining normal activities; 70-84 points: preventive rehabilitation medical gymnastics guidance training, regular review; 55-69 points: need brace treatment and rehabilitation training; 40-54 points: active brace treatment and rehabilitation intervention; <40 points: consider surgical treatment evaluation. Through such an evaluation system, the patient's scoliosis condition can be objectively quantified, providing a scientific basis for clinical treatment decisions, and also facilitating follow-up observation and efficacy evaluation.

[0038] In a preferred embodiment of the present invention, Figure 2 As shown, the plantar pressure signal is collected by the pressure sensor array; the surface electromyography signal is collected by the wireless multi-lead surface electrode; and the ultrasound image is collected by the high-frequency ultrasound probe. Then the time synchronization of the collected multimodal data is ensured by data synchronization control. Then the acquired signal is preprocessed, as shown in the following figure, and feature extraction is performed to obtain plantar pressure features, surface electromyography features and ultrasound image features. The plantar pressure features, surface electromyography features and ultrasound image features are input into the preset scoliosis feature fusion model for feature fusion to obtain static fusion features ( Figure 2 static evaluation indicators in) and dynamic fusion features ( Figure 2 The dynamic evaluation indicators in the evaluation are then used to calculate the comprehensive evaluation score. The scoliosis grade is determined based on the comprehensive evaluation score, and then the corresponding treatment decision is made, an evaluation report is generated, and more detailed intervention treatment is carried out based on the evaluation report.

[0039] In a preferred embodiment of the present invention, a patient with mild scoliosis is evaluated. Patient information: Gender: female; Age: 13 years old; Main complaint: suspected scoliosis was found in physical examination, and the full-length spinal X-ray showed an "S"-shaped scoliosis, the main curve Cobb angle was 15°, and the compensatory curve Cobb angle was 8°; Evaluation results: (1) Static fusion features (weight 40%): Plantar pressure signal analysis: left and right foot pressure symmetry index: 0.92 (normal ≥ 0.90); pressure center offset: 2.1 cm (slight offset); score: 85 points. Surface electromyography signal analysis: bilateral paraspinal muscle activity symmetry: 0.88 (normal ≥ 0.85); electromyography spectrum characteristics are normal; score: 82 points. Ultrasound image analysis: bilateral erector spinae muscle thickness difference: <10%; tissue elasticity is basically symmetrical; score: 88 points. Static feature evaluation score: (85+82+88) / 3×0.4=85 points×0.4=34 points.

[0040] (2) Dynamic fusion features (weight 60%): Gait analysis: gait cycle parameters are basically symmetrical; hip / knee / ankle joint range of motion is normal; score: 86 points. Dynamic electromyography analysis: muscle coordination pattern is slightly abnormal; integrated electromyography value changes moderately; score: 80 points. Dynamic foot pressure analysis: pressure center trajectory is slightly offset; footprint changes are within the normal range; score: 82 points. Dynamic feature evaluation score: (86+80+82) / 3×0.6=83 points×0.6=49.8 points.

[0041] (3) Comprehensive assessment score: 83.8 points (34+49.8); Scoliosis grade: good; Recommended measures: Standardized medical gymnastics training and regular follow-up observation are recommended.

[0042] In another preferred embodiment of the present invention, corresponding treatment measures are taken according to the evaluation report for the target to be evaluated. The basic information of the child is as follows: gender: female; age: 14 years old; height: 162 cm; weight: 48 kg; initial diagnosis Cobb angle: 28° (thoracic segment T6-T11) Analysis of initial assessment results (1) Static assessment: Plantar pressure signal analysis: left and right foot pressure symmetry index: 0.76; pressure center rightward 3.8 cm; score: 68 points. Surface electromyography signal analysis: right paraspinal muscle overactivation; left surface electromyography activity weakened; bilateral symmetry index: 0.71; score: 65 points. Ultrasound image analysis: convex erector spinae muscle thickness increased by 22%; concave muscle echo enhancement; tissue elasticity asymmetry; score: 63 points. Static feature assessment score: 65.3 points × 0.4 = 26.1 points.

[0043] (2) Dynamic assessment: Gait analysis: asymmetry of step length: 15%; limited range of motion of hip joint; score: 67 points. Dynamic electromyography analysis: abnormal muscle coordination during the walking cycle; continuous excessive contraction of convex side muscles; score: 64 points. Dynamic plantar pressure analysis: deviation of pressure center trajectory during walking; uneven weight bearing in the support phase; score: 66 points. Dynamic feature assessment score: 65.7 points × 0.6 = 39.4 points.

[0044] (3) Comprehensive evaluation score: 65.5 points (26.1+39.4): Scoliosis grade: General. Then, based on the comprehensive evaluation score and scoliosis grade, the initial treatment plan was generated: 1) Brace treatment: Schroth brace or Boston brace, etc.; wear 18-22 hours a day; adjust every 3 months; 2) Rehabilitation training program: paraspinal muscle balance training: 30 minutes / time, 2 times / day; core stability training: 40 minutes / time, 1 time / day 3) Schroth posture correction training: 20 minutes / time, 3 times / day. Treatment process monitoring and program optimization: 1) First review (3 months): Evaluation results: Comprehensive evaluation score: 69.8 points; Cobb angle: 26°; myoelectric symmetry improved; pressure distribution became more uniform. Program optimization: maintain the brace wearing time; increase proprioceptive function training; adjust the core training intensity, etc. 2) Second review (6 months): Evaluation results: Comprehensive evaluation score: 75.2 points; Cobb angle: 23°; Dynamic balance significantly improved; Muscle function and coordination enhanced. Program optimization: Brace wearing time adjusted to 16-17 hours; Add functional training programs; Add sports rehabilitation programs, etc. 3) Third review (9 months): Evaluation results: Comprehensive evaluation score: 82.5 points; Cobb angle: 20°; Biomechanical parameters are close to normal. Final optimization plan: Brace wearing time reduced to 12-14 hours; Transition to maintenance training; Develop a long-term exercise plan.

[0045] Among them, the key indicators for treatment plan optimization are: 1) Improvement of biomechanical indicators: symmetry of plantar pressure increased by ≥15%; symmetry of electromyographic activity increased by ≥20%; difference in tissue elasticity decreased by ≥25%, etc.; 2) Functional improvement indicators: normalization of gait parameters; improvement of motor control ability; improvement of daily activities, etc.; 3) Clinical indicators: reduction of Cobb angle by ≥5°; improvement of trunk balance; improvement of rotational deformity, etc. Through such dynamic optimization, the treatment effect can be monitored in real time, the treatment plan can be adjusted in time, treatment complications can be prevented, treatment compliance can be improved, and treatment prognosis can be improved.

[0046] In another preferred embodiment of the present invention, a 15-year-old female AIS patient, anthropometric data: height: 165cm; weight: 52kg; BMI: 19.1; bone age: Risser grade 3. Radiological data: S-type, main curve Cobb angle: 32° (T5-T12); compensatory curve: 18° (L1-L4); spinal rotation: Nash-Moe grade II; pelvic tilt: left high 2.1cm. Biomechanical parameters: paraspinal muscle mechanical properties; spinal flexibility index; trunk balance parameters, etc. Reconstruct a three-dimensional model of the spine based on CT / MRI; construct vertebral body and intervertebral disc units; establish a muscle-bone connection relationship; B. Material property definition: vertebral body-anisotropic elastic modulus; intervertebral disc-nonlinear hyperelasticity; ligament-nonlinear spring unit; muscle-Hill type muscle model; set gravity loading conditions; muscle contraction force and joint motion constraints, etc. To apply the prognostic prediction model, first input the parameter settings: basic information of the patient; clinical examination results; biomechanical parameters; treatment plan data, etc.; input prediction indicators: A. Short-term indicators (3-6 months): changes in Cobb angle; improvement in trunk balance; degree of muscle strength recovery; improvement in quality of life, etc.; B. Medium-term indicators (1-2 years): risk of spinal deformity progression; effect of brace treatment; degree of functional recovery; risk of complications, etc.; C. Long-term indicators (>2 years): terminal growth prediction; risk of deformity recurrence; quality of life expectations, etc.

[0047] Patient treatment plan formulation and prognosis prediction: A. Clinical manifestations: right thoracic convex deformity; shoulder asymmetry; mild trunk deviation, etc.; B. Biomechanical evaluation: excessive activation of paraspinal muscles on the convex side; rightward deviation of the pressure center; abnormal gait pattern, etc.

[0048] Model analysis: A. Biomechanical model analysis: Spinal stress distribution: Concave vertebral pressure increased by 23%; Concave ligament tension increased by 31%; Intervertebral disc shear stress abnormalities, etc. Muscle function analysis: Concave muscle fatigue index increased; Coordination mode disorder; Compensatory contraction mode, etc.

[0049] Prediction model analysis: brace treatment prediction: 6-month Cobb angle improvement expectation: 5-8°; trunk balance improvement probability: 75%; muscle strength recovery expectation: 60%-80%. Rehabilitation training prediction: myoelectric symmetry improvement: 25%-30%; gait parameter normalization: 70%; quality of life improvement: 65%, etc.

[0050] Personalized program formulation: A. Brace prescription optimization, type: Schroth brace or Boston brace; correction force setting: main point pressure: 35mmHg; secondary correction point: 25mmHg; wearing plan: initial-20-22 hours / day; decremental plan-evaluation every 3 months. B. Rehabilitation training plan, core stability training: intensity-medium; frequency-3 times / week; progressive plan-4 stages; special training: muscle strength balance training; posture control training; functional training, etc. Then regularly evaluate indicators: changes in biomechanical parameters; improvement in clinical symptoms; imaging changes; functional status score, etc.

[0051] Prediction model accuracy evaluation: deviation between predicted value and actual value <10%; prediction accuracy of key nodes >85%; risk warning accuracy >90%, etc. Through such personalized models and prediction systems, it is possible to achieve precise treatment plans, improve the accuracy of prognosis assessment, promote the optimization of treatment effects, and provide objective basis for clinical decision-making.

[0052] Specifically, plantar pressure signal measurement: Interface type: USB 3.0; Data transmission rate: ≥480Mbps Data format: {"timestamp":"YYYY-MM-DD HH:mm:ss.fff", "sensor_matrix":[[float]], "sampling_rate":int, "unit":"N / cm 2 "} Surface electromyography signal measurement: Interface type: Bluetooth 5.0; Transmission distance: ≥10m; Data format: {"timestamp":"YYYY-MM-DD HH:mm:ss.fff", "channel_id":int, "emg_data":[float], "sampling_rate":int, "unit":"mV"} Ultrasound image acquisition: Interface type: HDMI+USB 3.0; Image transmission: Real-time video stream; Data format: {"timestamp":"YYYY-MM-DD HH:mm:ss.fff", "image_data":"base64_encoded_string", "resolution":{"width":int,"height":int}, "depth":int, "gain":int} Data acquisition interface class DataAcquisitionInterface: def start_acquisition(self): """Start data collection""" def stop_acquisition(self): """Stop data collection""" def get_real_time_data(self): """Get real-time data""" def save_data(self): """Save collected data""" Data processing interface class DataProcessingInterface: def preprocess_data(self,raw_data): """Data preprocessing""" def extract_features(self,processed_data): """Feature extraction""" def fusion_analysis(self,features): """Fusion Analysis""" Evaluation decision interface class AssessmentInterface: def calculate_indicators(self,fusion_data): """Calculate evaluation metrics""" def generate_report(self,indicators): """Generate evaluation report""" def recommend_treatment(self,assessment_results): """Treatment plan recommendation""" By implementing this embodiment, the plantar pressure signal, surface electromyography signal and ultrasonic image of the target to be evaluated are obtained; the pressure center symmetry coefficient, the front and rear foot weight ratio and the lateral balance coefficient are extracted according to the plantar pressure signal to obtain the static characteristics of the plantar pressure; the pressure center movement trajectory, gait cycle parameters and joint movement trajectory are extracted to obtain the dynamic characteristics of the plantar pressure; the electromyography median frequency, electromyography average power frequency, electromyography average amplitude and electromyography spectrum area are extracted according to the surface electromyography signal to obtain the surface electromyography static characteristics; the integral electromyography value change rate and the electromyography signal change cycle are extracted to obtain the surface electromyography dynamic characteristics; the bilateral paravertebral skin thickness, fascia thickness, erector spinae transverse muscle thickness are extracted according to the ultrasonic image. The cross-sectional area, erector spinae circumference and elastic coefficient of muscle tissue are used to obtain the static features of ultrasound images; the tissue strain rate and tissue elastic modulus are extracted to obtain the dynamic features of ultrasound images; the static features of plantar pressure, the dynamic features of plantar pressure, the static features of surface electromyography, the dynamic features of surface electromyography, the static features of ultrasound images and the dynamic features of ultrasound images are input into the preset scoliosis feature fusion model for feature fusion to obtain static fusion features and dynamic fusion features; the static fusion features, the dynamic fusion features and the preset weight coefficients are used for calculation to obtain a comprehensive evaluation score; the scoliosis grade of the target to be evaluated is determined according to the comprehensive evaluation score and the preset scoliosis grade table. The data from three different sources, plantar pressure signals, surface electromyography signals and ultrasound images, are combined to reflect different aspects of human body movement, muscle activity and internal structure. The features from different sources are fused to obtain static fusion features and dynamic fusion features, which can more comprehensively evaluate the health of the spine, thereby improving the accuracy of scoliosis evaluation.

[0053] See also Figure 3 , is a schematic diagram of the structure of a scoliosis assessment system based on multimodal data provided by an embodiment of the present invention, comprising: A data acquisition module, used to acquire the plantar pressure signal, surface electromyography signal and ultrasonic image of the target to be evaluated; The first feature extraction module is used to extract the pressure center symmetry coefficient, the front and rear foot weight ratio and the lateral balance coefficient according to the plantar pressure signal to obtain the static characteristics of the plantar pressure; extract the pressure center movement trajectory, gait cycle parameters and joint movement trajectory to obtain the dynamic characteristics of the plantar pressure; The second feature extraction module is used to extract the electromyographic median frequency, the electromyographic average power frequency, the electromyographic average amplitude and the electromyographic spectrum area according to the surface electromyographic signal to obtain the surface electromyographic static features; extract the integral electromyographic value change rate and the electromyographic signal change period to obtain the surface electromyographic dynamic features; The third feature extraction module is used to extract the bilateral paravertebral skin thickness, fascia thickness, erector spinae muscle cross-sectional area, erector spinae muscle circumference and muscle tissue elasticity coefficient according to the ultrasound image to obtain the static features of the ultrasound image; extract the tissue strain rate and tissue elastic modulus to obtain the dynamic features of the ultrasound image; A feature fusion module, for inputting the static feature of plantar pressure, the dynamic feature of plantar pressure, the static feature of surface electromyography, the dynamic feature of surface electromyography, the static feature of ultrasound image and the dynamic feature of ultrasound image into a preset scoliosis feature fusion model for feature fusion, so as to obtain static fusion features and dynamic fusion features; A comprehensive evaluation module, used to calculate according to the static fusion feature, the dynamic fusion feature and a preset weight coefficient to obtain a comprehensive evaluation score; The scoliosis grade determination module is used to determine the scoliosis grade of the target to be evaluated based on the comprehensive evaluation score and a preset scoliosis grade table.

[0054] The present invention provides a scoliosis assessment system based on multimodal data. According to a data acquisition module, a plantar pressure signal, a surface electromyography signal and an ultrasonic image of a target to be assessed are acquired; in a first feature extraction module, a pressure center symmetry coefficient, a front and rear foot weight ratio and a lateral balance coefficient are extracted according to the plantar pressure signal to obtain a plantar pressure static feature; a pressure center movement trajectory, a gait cycle parameter and a joint movement trajectory are extracted to obtain a plantar pressure dynamic feature; in a second feature extraction module, electromyography median frequency, electromyography average power frequency, electromyography average amplitude and electromyography spectrum area are extracted according to the surface electromyography signal to obtain a surface electromyography static feature; an integrated electromyography value change rate and an electromyography signal change cycle are extracted to obtain a surface electromyography dynamic feature; in a third feature extraction module, bilateral paravertebral skin thickness is extracted according to the ultrasonic image. The static features of ultrasound images are obtained by extracting the strain rate and elastic modulus of tissues; the dynamic features of ultrasound images are obtained by extracting the plantar pressure static features, the plantar pressure dynamic features, the surface electromyography static features, the surface electromyography dynamic features, the ultrasound image static features and the ultrasound image dynamic features into the preset scoliosis feature fusion model through the feature fusion module to obtain static fusion features and dynamic fusion features; then in the comprehensive evaluation module, the static fusion features, the dynamic fusion features and the preset weight coefficients are calculated to obtain a comprehensive evaluation score; finally, in the scoliosis grade determination module, the scoliosis grade of the target to be evaluated is determined according to the comprehensive evaluation score and the preset scoliosis grade table. The data from three different sources, namely, plantar pressure signals, surface electromyography signals and ultrasound images, are combined to reflect different aspects of human movement, muscle activity and internal structure. The features from different sources are fused to obtain static fusion features and dynamic fusion features, which can more comprehensively evaluate the health of the spine, thereby improving the accuracy of scoliosis evaluation.

[0055] It should be noted that the system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the system embodiment provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0056] Those skilled in the art can clearly understand that, for the sake of convenience and brevity, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0057] Another embodiment of the present invention further provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements a scoliosis assessment method based on multimodal data as described in the above embodiment when executing the computer program. The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The terminal device may include, but is not limited to, a processor and a memory.

[0058] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.

[0059] The memory can be used to store the computer program, and the processor realizes various functions of the terminal device by running or executing the computer program stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device or other volatile solid-state storage device.

[0060] Another embodiment of the present invention provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a scoliosis assessment method based on multimodal data as described in the above embodiment.

[0061] The storage medium is a computer-readable storage medium, and the computer program is stored in the computer-readable storage medium. When the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or system that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0062] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A scoliosis assessment method based on multimodal data, characterized in that: include: Acquire plantar pressure signals, surface electromyography signals and ultrasound images of the target to be evaluated; Extract the pressure center symmetry coefficient, the front and rear foot weight ratio and the lateral balance coefficient according to the plantar pressure signal to obtain the static characteristics of the plantar pressure; extract the pressure center movement trajectory, gait cycle parameters and joint movement trajectory to obtain the dynamic characteristics of the plantar pressure; Extracting the electromyographic median frequency, the electromyographic average power frequency, the electromyographic average amplitude and the electromyographic spectrum area according to the surface electromyographic signal to obtain the surface electromyographic static characteristics; Extract the integral EMG value change rate and EMG signal change cycle to obtain the surface EMG dynamic characteristics; Based on the ultrasound images, the bilateral paravertebral skin thickness, fascia thickness, erector spinae muscle cross-sectional area, erector spinae muscle circumference and muscle tissue elasticity coefficient were extracted to obtain the static features of the ultrasound images. Extract tissue strain rate and tissue elastic modulus to obtain dynamic characteristics of ultrasound images; Inputting the static features of plantar pressure, the dynamic features of plantar pressure, the static features of surface electromyography, the dynamic features of surface electromyography, the static features of ultrasound images, and the dynamic features of ultrasound images into a preset scoliosis feature fusion model for feature fusion to obtain static fusion features and dynamic fusion features; Calculate according to the static fusion feature, the dynamic fusion feature and a preset weight coefficient to obtain a comprehensive evaluation score; The scoliosis grade of the target to be evaluated is determined based on the comprehensive evaluation score and a preset scoliosis grade table.

2. A scoliosis assessment method based on multimodal data as claimed in claim 1, characterized in that: The training of the scoliosis feature fusion model includes: Acquire historical plantar pressure signals, historical surface electromyography signals, historical ultrasound images, historical static fusion features, historical dynamic fusion features and initial model parameters of the scoliosis target; Extract the pressure center symmetry coefficient, the front and rear foot weight ratio and the lateral balance coefficient according to the historical plantar pressure signal to obtain the historical plantar pressure static characteristics; extract the pressure center movement trajectory, gait cycle parameters and joint movement trajectory to obtain the historical plantar pressure dynamic characteristics; Extract the median EMG frequency, the average EMG power frequency, the average EMG amplitude and the EMG spectrum area according to the historical surface EMG signal to obtain the historical surface EMG static characteristics; extract the integral EMG value change rate and the EMG signal change period to obtain the historical surface EMG dynamic characteristics; According to the historical ultrasound images, the bilateral paravertebral skin thickness, fascia thickness, erector spinae muscle cross-sectional area, erector spinae muscle circumference and muscle tissue elastic coefficient are extracted to obtain the static features of the historical ultrasound images; the tissue strain rate and tissue elastic modulus are extracted to obtain the dynamic features of the historical ultrasound images; Input the historical plantar pressure static features, historical plantar pressure dynamic features, historical surface electromyography static features, historical surface electromyography dynamic features, historical ultrasound image static features, and historical ultrasound image dynamic features into the scoliosis feature fusion model to be trained, perform iterative training, and obtain predicted static fusion features and predicted dynamic fusion features; Calculating a loss function value of a preset model loss function according to the predicted static fusion feature, the predicted dynamic fusion feature, the historical static fusion feature, and the historical dynamic fusion feature; When the loss function value is determined to be convergent, a trained scoliosis feature fusion model is obtained; When it is determined that the loss function value has not converged, the current model parameters are updated according to the model loss function; and the updated current model parameters are used as the current model parameters for the next training; Among them, the current model parameters at the initial time are the initial model parameters.

3. A scoliosis assessment method based on multimodal data as claimed in claim 2, characterized in that: Extracting the pressure center symmetry coefficient, the front and rear foot weight ratio, and the lateral balance coefficient according to the plantar pressure signal to obtain a plantar pressure static feature; Extract the pressure center movement trajectory, gait cycle parameters and joint movement trajectory to obtain the dynamic characteristics of the plantar, including: Determining the left foot load pressure point distribution and the right foot load pressure point distribution according to the plantar pressure signal; The pressure center symmetry coefficient is calculated according to the left foot load pressure point distribution and the right foot load pressure point distribution; the forefoot pressure sum and the rear foot pressure sum are calculated according to the left foot load pressure point distribution and the right foot load pressure point distribution respectively; and the forefoot and rearfoot weight ratio is calculated according to the forefoot pressure sum and the rear foot pressure sum; the plantar lateral pressure sum and the plantar medial pressure sum are calculated according to the left foot load pressure point distribution and the right foot load pressure point distribution respectively; and the plantar lateral balance coefficient is calculated according to the plantar lateral pressure sum and the plantar medial pressure sum; The pressure center symmetry coefficient, the front and rear foot weight ratio and the lateral balance coefficient are used as static characteristics of plantar pressure; Determine the coordinate change of the pressure center according to the plantar pressure signal to obtain the movement trajectory of the pressure center; Determining the starting point and the ending point of the gait cycle according to the plantar pressure signal; Calculate the gait cycle duration, the stance phase duration, and the swing phase duration according to the starting point and the end point to obtain gait cycle parameters; Extracting joint motion trajectory according to the plantar pressure signal; The pressure center movement trajectory, the gait cycle parameters and the joint movement trajectory are used as dynamic features of plantar pressure.

4. A scoliosis assessment method based on multimodal data as claimed in claim 3, characterized in that: Extracting the electromyographic median frequency, the electromyographic average power frequency, the electromyographic average amplitude and the electromyographic spectrum area according to the surface electromyographic signal to obtain the surface electromyographic static characteristics; Extract the integral EMG value change rate and the EMG signal change cycle to obtain the surface EMG dynamic characteristics, including: Performing Fourier transform on the surface electromyography signal to obtain an electromyography spectrum diagram; According to the electromyographic spectrum, determining the electromyographic median frequency, the electromyographic average power frequency, the electromyographic average amplitude and the electromyographic spectrum area; The EMG median frequency, the EMG average power frequency, the EMG average amplitude and the EMG spectrum area are used as surface EMG static features; Performing segmented processing according to the surface electromyographic signal and a preset time window to obtain a plurality of segmented electromyographic signals; According to the segmented electromyographic signals, the integrated electromyographic value of each segmented electromyographic signal is calculated; and according to the integrated electromyographic value, a calculation is performed to obtain the integrated electromyographic value change rate; Performing time-frequency analysis on the surface electromyographic signal to extract the electromyographic signal variation period; The integrated electromyographic value change rate and the electromyographic signal change period are used as surface electromyographic dynamic characteristics.

5. A scoliosis assessment method based on multimodal data as claimed in claim 4, characterized in that: Based on the ultrasound images, the bilateral paravertebral skin thickness, fascia thickness, erector spinae muscle cross-sectional area, erector spinae muscle circumference and muscle tissue elasticity coefficient were extracted to obtain the static features of the ultrasound images. Extract tissue strain rate and tissue elastic modulus to obtain dynamic features of ultrasound images, including: Determine the skin layer, fascia layer and erector spinae muscle contour according to the ultrasound image; Determine the thickness of the skin layer according to the skin layer; and calculate the bilateral paravertebral skin thickness according to the thickness of the skin layer; determining a fascia thickness according to the fascia layer; According to the erector spinae muscle contour, the cross-sectional area of ​​the erector spinae muscle and the circumference of the erector spinae muscle are calculated; Determine the elasticity distribution map of the muscle tissue according to the ultrasonic image; and calculate the elasticity coefficient of the muscle tissue according to the elasticity distribution map; The bilateral paravertebral skin thickness, fascia thickness, erector spinae muscle cross-sectional area, erector spinae muscle circumference and muscle tissue elasticity coefficient are used as static features of ultrasound images; Determining tissue movement velocity and tissue deformation parameters according to the ultrasonic image; Calculating the tissue strain rate according to the tissue movement speed; Calculating the tissue elastic modulus according to the tissue deformation parameter; The tissue strain rate and the tissue elastic modulus are used as dynamic features of ultrasound images.

6. A scoliosis assessment method based on multimodal data as claimed in claim 5, characterized in that: The weight coefficients include static feature weight coefficients and dynamic feature weight coefficients; The static fusion feature, the dynamic fusion feature and the preset weight coefficient are used to calculate and obtain a comprehensive evaluation score, including: Performing normalization processing according to the static fusion feature and the dynamic fusion feature to obtain a target static fusion feature and a target dynamic fusion feature; The static feature evaluation score is calculated by multiplying the target static fusion feature and the static feature weight coefficient; the dynamic feature evaluation score is calculated by multiplying the target dynamic fusion feature and the dynamic feature weight coefficient; the comprehensive evaluation score is calculated by adding the static feature evaluation score and the dynamic feature evaluation score.

7. A scoliosis assessment method based on multimodal data as claimed in claim 6, characterized in that: The scoliosis grade table includes: assessment score intervals and scoliosis grades; Determining the scoliosis grade of the target to be evaluated according to the comprehensive evaluation score and a preset scoliosis grade table, including: traversing the evaluation score interval in the preset scoliosis grade table according to the comprehensive evaluation score to determine the corresponding evaluation score interval; According to the corresponding assessment score range, the corresponding scoliosis grade is determined.

8. The method for assessing scoliosis based on multimodal data as claimed in claim 7, characterized in that: Also includes: Determine corresponding treatment measures according to the scoliosis grade; An assessment report is generated based on the comprehensive assessment score, the scoliosis grade and the corresponding treatment measures, so that corresponding treatment measures are taken for the assessment target according to the assessment report.

9. The method for assessing scoliosis based on multimodal data as claimed in claim 8, characterized in that: Also includes: The plantar pressure signal is collected by a pressure sensor array; The surface electromyography signal is collected by wireless multi-lead surface electrodes; The ultrasonic image is collected by a high-frequency ultrasonic probe.

10. A scoliosis assessment system based on multimodal data, characterized in that: include: A data acquisition module, used to acquire the plantar pressure signal, surface electromyography signal and ultrasonic image of the target to be evaluated; The first feature extraction module is used to extract the pressure center symmetry coefficient, the front and rear foot weight ratio and the lateral balance coefficient according to the plantar pressure signal to obtain the static characteristics of the plantar pressure; extract the pressure center movement trajectory, gait cycle parameters and joint movement trajectory to obtain the dynamic characteristics of the plantar pressure; A second feature extraction module is used to extract the electromyography median frequency, the electromyography average power frequency, the electromyography average amplitude and the electromyography spectrum area according to the surface electromyography signal to obtain the surface electromyography static features; Extract the integral EMG value change rate and EMG signal change cycle to obtain the surface EMG dynamic characteristics; The third feature extraction module is used to extract the bilateral paravertebral skin thickness, fascia thickness, erector spinae muscle cross-sectional area, erector spinae muscle circumference and muscle tissue elasticity coefficient according to the ultrasound image to obtain the static features of the ultrasound image; Extract tissue strain rate and tissue elastic modulus to obtain dynamic characteristics of ultrasound images; A feature fusion module, for inputting the static feature of plantar pressure, the dynamic feature of plantar pressure, the static feature of surface electromyography, the dynamic feature of surface electromyography, the static feature of ultrasound image and the dynamic feature of ultrasound image into a preset scoliosis feature fusion model for feature fusion, so as to obtain static fusion features and dynamic fusion features; A comprehensive evaluation module, used to calculate according to the static fusion feature, the dynamic fusion feature and a preset weight coefficient to obtain a comprehensive evaluation score; The scoliosis grade determination module is used to determine the scoliosis grade of the target to be evaluated based on the comprehensive evaluation score and a preset scoliosis grade table.

Images