Sasm and contracture evaluation method and system based on multi-scale data feature fusion

Abstract

The invention relates to the technical field of rehabilitation evaluation, in particular to a spasm and contracture evaluation method and system based on multi-scale data feature fusion. The method comprises the following steps: collecting data such as biomechanics, surface myoelectricity, muscle-bone ultrasound and clinical information of patients suffering from spasm and contracture after clinical cerebral apoplexy; preprocessing the data, wherein the preprocessing comprises chain equation multiple filling, label coding, normalization processing, feature splicing, data transformation and data enhancement; constructing a deep learning model based on the combination of a CNN network, multi-scale feature fusion and a self-attention mechanism, wherein the deep learning model sequentially comprises a first feature extraction module, a second feature extraction module, a third feature extraction module, a feature fusion module and a classification module; inputting the preprocessed data into the model for training, and obtaining a spasm evaluation deep learning model through multi-round iteration optimization of model parameters; and inputting to-be-evaluated data into the model, and outputting an evaluation result. The method has the advantages that local features are effectively captured, the global dependency relationship is captured, and the model classification and generalization ability is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of rehabilitation evaluation, and in particular to a method and system for evaluating spasticity and contracture based on multi-scale data feature fusion. Background Art

[0002] Spasticity and contracture are common sequelae after stroke. Among them, spasticity, characterized by increased resistance to passive movement and high muscle tension, can lead to muscle stiffness, pain or disuse, etc. Contracture is caused by changes in the structure of muscle fibers and connective tissues associated with lack of movement and long-term spasticity, leading to a reduction in the range of joint motion and flexion contracture deformity, which seriously affects the functional independence and quality of life of patients.

[0003] Accurate functional assessment is crucial for the formulation of motor function rehabilitation plans after stroke and for the observation of therapeutic effects. However, the commonly used clinical assessment methods are mostly subjective rating scales, which lack objectivity and cannot obtain accurate physiological information and motor function status of the human body. They have limited ability to distinguish between neural components and biomechanical components, making it difficult to obtain the best treatment plan. Currently, commonly used objective methods include neuroelectrophysiology, biomechanics, imaging, and some research on human-machine engineering related devices. These methods use instruments to quantitatively measure and evaluate nerve and muscle related parameters, which are more objective and reliable than clinical scales. However, at present, there is still no unified and recognized gold standard for the above methods.

[0004] Therefore, in the context of precision medicine, how to more accurately evaluate spasticity and contracture in stroke patients is a key issue that needs to be urgently addressed in this field. Summary of the invention

[0005] In order to solve the above problems, the present invention provides a method and system for evaluating spasm and contracture based on multi-scale data feature fusion.

[0006] The first object of the present invention is to provide a method for evaluating spasm and contracture based on multi-scale data feature fusion, which specifically comprises the following steps: S1. Collect biomechanical data, surface electromyography data, musculoskeletal ultrasound data, and clinical information data of patients with post-stroke spasticity and contracture; record the assessment levels of the modified Ashworth grading scale and the modified Tardieu grading scale corresponding to the patients as label data; S2. Preprocessing the data collected in step S1, including chain equation multiple filling, label encoding, normalization, feature splicing, data transformation and data enhancement; S3. Constructing a deep learning model based on a CNN network, multi-scale feature fusion and self-attention mechanism; the deep learning model includes a first feature extraction module, a second feature extraction module, a third feature extraction module, a feature fusion module and a classification module in sequence; S4. Input the preprocessed data into the constructed deep learning model for training, optimize the model parameters through multiple rounds of iterations, and obtain a deep learning model for spasticity evaluation; S5. Input the data to be evaluated into the spasm evaluation deep learning model, and output the evaluation results of spasm and contracture.

[0007] Preferably, in step S1, a stratified sampling method is used for data collection; The biomechanical data are indicators of the normal side and the abnormal side measured at 8 angular velocities, each side has 6 indicator items, and the shape of a single data item is 8 12; The surface electromyography data are indicators of the normal side and the abnormal side measured at 8 angular velocities. Each side has 12 indicator items, and the shape of a single data item is 8 twenty four; The musculoskeletal ultrasound data are divided into normal side and abnormal side, each side has 10 indicators, and the shape of a single data is 1 20; The clinical information data covers the basic information of the patient, with 10 indicators, and the shape of a single data item is 1 10; The modified Ashworth grading scale is used to assess the patient's muscle tension level. Different grades correspond to different degrees of increased muscle tension. The shape of a single data item is 1 1; The modified Tardieu grading scale is used to assess the degree of spasticity. Different grades correspond to different degrees of spasticity severity. The shape of a single data item is 1 1.

[0008] Preferably, the data transformation in step S2 includes rotation splicing and mirroring, and the data enhancement is Mixup data mixing; specifically, it includes the following sub-steps: S201. Chain equation multiple imputation: MICE multiple imputation method was used to construct MICE models based on biomechanical data, surface electromyography data, musculoskeletal ultrasound data and clinical information data, and the data with missing values ​​in some columns were imputed; S202. Label encoding: The modified Ashworth grading scale is mapped into 6 categories, where 0, 1, 1+, 2, 3, and 4 correspond to 0, 1, 2, 3, 4, and 5 respectively; the modified Tardieu grading scale is mapped into 6 categories, where 0, 1, 2, 3, 4, and 5 correspond to 0, 1, 2, 3, 4, and 5 respectively; S203. Data pairing: biomechanical data, surface electromyography data, musculoskeletal ultrasound data, clinical information data, and the assessment levels of the modified Ashworth grading scale and the modified Tardieu grading scale are paired and combined according to the patient's name to form a standardized data set; S204. Normalization processing: after pairing, the maximum and minimum values ​​of the biomechanical data, surface electromyography data, musculoskeletal ultrasound data, and clinical information data in the column direction are calculated, and the maximum and minimum values ​​are used for normalization in the row direction; S205. Feature splicing processing: After normalization processing, the biomechanical data, the musculoskeletal ultrasound data and the clinical information data are padded with 0 using the surface electromyography data as the reference width, and the biomechanical data, the surface electromyography data, the musculoskeletal ultrasound data and the clinical information data are spliced ​​in the 0th dimension to form a matrix; S206. Rotation splicing: Splice the matrix 8 times to form 54 Matrix of 72; S207. Mirror processing: first flip the spliced ​​matrix horizontally, and then flip it vertically; S208.Mixup data mixing: Randomly select two matrices from the matrix set processed by S207, mix the two matrices and labels by linear interpolation, and assign mixing weights to different categories.

[0009] Preferably, step S204 uses a Min-Max variant to normalize the data, which is expressed as:

[0010] in μ represents the mean; σ represents the standard deviation; a represents the mean minus one standard deviation; b represents the mean plus one standard deviation; X represents the unprocessed observation; X norm represents the normalized data; X max Indicates the maximum value; X min Indicates the minimum value; The step S205 specifically includes: the biomechanical data size is 8 12. The first six columns are abnormal sides, and the last six columns are normal sides. Six zeros are added to the end of the six abnormal side indicators, and six zeros are added to the end of the six normal side indicators. The shape after filling is 8. 24; musculoskeletal ultrasound data size is 1 20, the first ten columns are abnormal side, the last ten columns are normal side, add 2 zeros after the ten abnormal side indicators, and add 2 zeros before the ten normal measurement indicators, the shape after filling is 1 24; add 14 zeros to the end of the clinical information data, and the shape after filling is 1 24; The biomechanical data, surface electromyography data, musculoskeletal ultrasound data and clinical information data are spliced ​​in the 0th dimension to form 18 Matrix of 24; In the matrix after rotation and splicing in step S206, the center position retains the original matrix, and other sub-matrices are truncated to varying degrees after rotation. After truncation, only local feature information is retained, and there is no time dimension information in rows and columns.

[0011] Preferably, the first feature extraction module includes two scales of convolution and self-attention mechanism for capturing low-level features and global dependencies; the second feature extraction module includes two scales of convolution and self-attention mechanism for capturing mid-level features and global dependencies; the third feature extraction module includes a single scale of convolution and self-attention mechanism for capturing high-level features and global dependencies; the feature fusion module includes an upsampling module, a jump connection module and a fusion convolution module for fusing low-level and high-level features; the classification module includes two fully connected classification layers for classification; the first feature extraction module, the second feature extraction module, the third feature extraction module and the feature fusion module sequentially output features fx of one scale, x=1, 2, 3, 4; and finally the classification module outputs a specific category; The convolutions of the two scales are 3 3 convolution kernels of low-level convolution and 5 5 high-level convolution of convolution kernel; 3 3 The low-level convolution of the convolution kernel is filled with 1, and the extracted low-level features contain rich detail information; 5 The high-level convolution of the 5-convolution kernel is padded to 2, and the extracted high-level features contain more semantic information; The single-scale convolution is 3 3 convolution kernels for low-level convolution.

[0012] Preferably, the calculation method of the self-attention mechanism is as follows: S301. Input feature vector X and map it to query matrix, key matrix and value matrix respectively through three linear transformations or fully connected layers: Q=XW Q K=XW K V=XW V Among them, WQ , W K and W V is a learnable weight matrix with dimensions d×d k , d k is the dimension of the key vector; Q, K, and V represent the query matrix, key matrix, and value matrix, respectively; the dimensions of the calculated query matrix, key matrix, and value matrix are all n×d k ; n is the sequence length, d is the feature dimension; S302. Calculate the attention score matrix based on the dot product of the query matrix Q and the key matrix K transpose:

[0013] Among them, K T is the transpose of the key matrix K; QK T is the dot product of the transpose of the query matrix Q and the key matrix K, with dimension n×n; S303. Use the Softmax function to normalize the scaled attention scores to obtain the attention weight matrix: ; The softmax function is calculated row by row to ensure that the sum of the weights of each row is 1, thereby converting the attention score matrix into a probability distribution, indicating the relative importance of each eigenvalue to other eigenvalues:

[0014] Among them, A ij represents the attention score of the i-th eigenvalue to the j-th eigenvalue; exp(A ij ) is A ij The exponential function value of n k=1 exp(A ik ) is the sum of the exponential values ​​of all elements in the i-th row; S304. Use the attention weight matrix to perform weighted sum on the value matrix V to obtain the final output feature:

[0015] Among them, AttentionWeights represents the attention weight matrix, the dimension is n×n, and the dimension of the value matrix V is n×d k , the dimension of the output feature is n×d k .

[0016] Preferably, the training strategy of step S4 is as follows: The addition of two cross entropy loss functions is used as the objective function, one of which is used to calculate the loss between the modified Ashworth grading scale rating level predicted by the model and the true modified Ashworth grading scale rating level label, and the other cross entropy loss function is used to calculate the loss between the modified Tardieu grading scale rating level predicted by the model and the true modified Tardieu grading scale rating level label; the Adam optimizer is used, the initial learning rate is set to 1e-3, the L2 norm regularization weight decay value is 5e-7, the exponential decay rate of the first-order moment estimate is 0.95, and the exponential decay rate of the second-order moment estimate is 0.999.

[0017] Preferably, in step S4, during the training process, gradient-weighted class activation mapping is used to calculate the average gradient of biomechanical data, surface electromyography data, musculoskeletal ultrasound data, and clinical information data, and the contribution of biomechanical data, surface electromyography data, musculoskeletal ultrasound data, and clinical information data to model decision-making is evaluated respectively according to the average gradient score, and the degree of dependence of the model on surface electromyography data, musculoskeletal ultrasound data, and clinical information data is explained; the evaluation indicators include accuracy, average precision, and area under the ROC curve.

[0018] Preferably, the biomechanical data include: maximum passive flexion angle of the ankle joint, maximum passive extension angle of the ankle joint, stretching torque, flexion torque, average power, and average work done; The specific indicators of the surface electromyography data include: medial gastrocnemius head-average electromyography value, lateral gastrocnemius head-average electromyography value, soleus muscle-average electromyography value, medial gastrocnemius head-root mean square value, lateral gastrocnemius head-root mean square value, soleus muscle-root mean square value, medial gastrocnemius head-median frequency, lateral gastrocnemius head-median frequency, soleus muscle-median frequency, medial gastrocnemius head-power frequency, lateral gastrocnemius head-power frequency, soleus muscle-power frequency; The specific indicators of the musculoskeletal ultrasound data include: fiber length, feather angle, medial gastrocnemius head - cross-sectional area, medial gastrocnemius head - muscle thickness, medial gastrocnemius head - shear wave velocity, lateral gastrocnemius head - cross-sectional area, lateral gastrocnemius head - muscle thickness, lateral gastrocnemius head - shear wave velocity, soleus muscle - muscle thickness, soleus muscle - shear wave velocity; Specific indicators of the clinical information data include: gender, age, height, weight, Barthel index, ankle passive joint range of motion, V1 speed limb reach angle, V3 speed stagnation angle, lower limb motor recovery stage, and lower limb motor function score.

[0019] The second object of the present invention is to provide a spasm and contracture evaluation system based on multi-scale data feature fusion, comprising: a data acquisition module, a data preprocessing module, a model building module, a model training module and an evaluation module; A data acquisition module is used to collect biomechanical data, surface electromyography data, musculoskeletal ultrasound data, clinical information data, and the assessment levels of the modified Ashworth grading scale and the modified Tardieu grading scale of clinical patients with post-stroke spasticity and contracture, and send the data to a data preprocessing module; The data preprocessing module is used to preprocess the data. The preprocessing includes chain equation multiple filling, label encoding, normalization, feature concatenation, data transformation and data enhancement. After preprocessing, the data is sent to the model training module. Model building module, used to build a deep learning model based on CNN network, multi-scale feature fusion and self-attention mechanism; A model training module is used to input the preprocessed data into the constructed deep learning model for training, and optimize the model parameters through multiple rounds of iterations to obtain a deep learning model for spasticity evaluation; The evaluation module is used to input the data to be evaluated into the trained model and output the evaluation results of spasticity and contracture.

[0020] Compared with the prior art, the present invention can achieve the following beneficial effects: The present invention provides a method and system for evaluating spasticity based on multi-scale data feature fusion. Through the integration and analysis of multi-source data (including biomechanical data, surface electromyography data, musculoskeletal ultrasound data, clinical information data, and the assessment levels of the modified Ashworth grading scale and the modified Tardieu grading scale), combined with the CNN model, multi-scale feature fusion and self-attention mechanism, a more accurate and objective method for evaluating spasticity and contracture is provided. This method can not only effectively capture local features, but also capture global dependencies through the self-attention mechanism, significantly improving the classification performance and generalization ability of the model. Compared with the traditional single data source evaluation method, the present invention can more comprehensively reflect the patient's physiological state and motor function. The application fields of the present invention include but are not limited to post-stroke rehabilitation evaluation, motor dysfunction diagnosis, and the formulation of personalized rehabilitation programs. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 The present invention provides a flowchart of a method for evaluating spasm and contracture based on multi-scale data feature fusion according to an embodiment of the present invention.

[0022] Figure 2 It is a schematic diagram of a single matrix feature provided according to an embodiment of the present invention.

[0023] Figure 3 Schematic diagram of matrix features after feature splicing provided by an embodiment of the present invention.

[0024] Figure 4It is a structural diagram of a deep learning model provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, the same modules are represented by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, the detailed description thereof will not be repeated.

[0026] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0027] The present invention provides a method for evaluating spasm and contracture based on multi-scale data feature fusion. Figure 1 , specifically including the following steps: S1. Collect biomechanical data, surface electromyography data, musculoskeletal ultrasound data, clinical information data, and the assessment levels of the modified Ashworth grading scale and the modified Tardieu grading scale of patients with post-stroke spasticity and contracture; specifically, the data were collected using a stratified sampling method; the data came from real data of patients with post-stroke spasticity and contracture who visited the Rehabilitation Department of the First Hospital of Jilin University; Biomechanical data is derived from the mechanical data collection of joints under different angular velocity passive motion states, which can accurately reflect the compliance or stiffness of the soft tissue around the joints. Biomechanical data are indicators of passive activities of the normal and abnormal ankle joints measured at 8 angular velocities. Each side has 6 indicators, and the shape of a single data item is 8. 12. Specific indicators include: maximum passive flexion angle of ankle joint, maximum passive extension angle of ankle joint, stretching torque, flexion torque, average power, and average work done.

[0028] Surface electromyography data is derived from the bioelectric signal collection of specific joints under different angular velocity motion states, which can accurately reflect the functional state and activity mode of the joints. Surface electromyography data is the index of the normal side and the abnormal side measured at 8 angular velocities. Each side has 12 index items, and the shape of a single data is 8 24. Specific indicators include: medial gastrocnemius head-average EMG value, lateral gastrocnemius head-average EMG value, soleus muscle-average EMG value, medial gastrocnemius head-root mean square value, lateral gastrocnemius head-root mean square value, soleus muscle-root mean square value, medial gastrocnemius head-median frequency, lateral gastrocnemius head-median frequency, soleus muscle-median frequency, medial gastrocnemius head-power frequency, lateral gastrocnemius head-power frequency, soleus muscle-power frequency.

[0029] Musculoskeletal ultrasound data uses ultrasound technology to measure specific muscle groups, reflecting the morphology and pathological conditions of the muscle groups. Musculoskeletal ultrasound data is divided into normal side and abnormal side, each side has 10 indicators, and the shape of a single data is 1 20. Specific indicators include: fiber length, pennation angle, medial gastrocnemius head - cross-sectional area, medial gastrocnemius head - muscle thickness, medial gastrocnemius head - shear wave velocity, lateral gastrocnemius head - cross-sectional area, lateral gastrocnemius head - muscle thickness, lateral gastrocnemius head - shear wave velocity, soleus muscle - muscle thickness, soleus muscle - shear wave velocity.

[0030] Clinical information data covers the basic information of patients, with 10 indicators, and the shape of a single data item is 1 10; Specific indicators include: gender, age, height, weight, Barthel index, ankle passive joint range of motion, V1 speed limb reach angle, V3 speed stagnation angle, lower limb motor recovery stage, and lower limb motor function score.

[0031] The modified Ashworth grading scale is used to assess the patient's muscle tension level. Different levels correspond to different degrees of increased muscle tension. The shape of a single data item is 1 1; The modified Tardieu grading scale is used to assess the degree of spasticity. Different grades correspond to different degrees of spasticity severity. The shape of a single data item is 1 1; S2. Preprocess the data collected in step S1, including chain equation multiple filling, label encoding, normalization, feature splicing, data transformation and data enhancement; specifically, data transformation includes rotation splicing and mirroring; data enhancement is Mixup data mixing; in this step, data pairing is also required before feature splicing; specifically, the following sub-steps are included: S201. Chain equation multiple imputation: MICE multiple imputation method was used to construct MICE models based on biomechanical data, surface electromyography data, musculoskeletal ultrasound data and clinical information data, and the data with missing values ​​in some columns were imputed; S202. Label encoding: The modified Ashworth grading scale is mapped into 6 categories, where 0, 1, 1+, 2, 3, and 4 correspond to 0, 1, 2, 3, 4, and 5 respectively; the modified Tardieu grading scale is mapped into 6 categories, where 0, 1, 2, 3, 4, and 5 correspond to 0, 1, 2, 3, 4, and 5 respectively; The evaluation levels of the modified Ashworth grading scale include: level 0 means no increase in muscle tension, and passive movement of the affected limb has resistance throughout the range of joint movement; level 1 means a slight increase in muscle tension, and there is slight resistance when passively moving the affected limb to the end of the joint range of motion; level 1+ means a slight increase in muscle tension, and there is a slight "stuck" feeling when passively moving the affected limb at 1 / 2 of the joint range of motion, and slight resistance in the posterior joint range of motion; level 2 means a slight increase in muscle tension, and there is resistance in most of the joint range of motion when passively moving the affected limb, but the affected range is still easy to move; level 3 means a moderate increase in muscle tension, and there is resistance in the entire range of joint movement when passively moving the affected limb, and movement is relatively difficult; level 4 means a severe increase in muscle tension, stiffness of the affected limb, greater resistance, and difficulty in passive movement; The evaluation levels of the modified Tardieu grading scale include: level 0 means no resistance during passive movement; level 1 means slight resistance is felt during passive movement, but the position is not determined; level 2 means resistance is suddenly felt at a precise angle during passive movement, and then decreases; level 3 means fatigue clonus occurs at a precise angle in the range of joint motion, that is, clonus <10 seconds under continuous stretching force, and then released; level 4 means non-fatiguing clonus occurs at a precise angle in the range of joint motion, that is, clonus >10 seconds under continuous stretching force; level 5 means that the joint cannot be passively moved; S203. Data pairing: biomechanical data, surface electromyography data, musculoskeletal ultrasound data, clinical information data, and the assessment levels of the modified Ashworth grading scale and the modified Tardieu grading scale are paired and combined according to the patient's name (ID) to form a standardized data set; S204. Normalization processing: After pairing, calculate the maximum value X of biomechanical data, surface electromyography data, musculoskeletal ultrasound data, and clinical information data in the column direction max and minimum value X min , use the maximum and minimum values ​​in the row direction for normalization; use the Min-Max variant to normalize the data, expressed as:

[0032] in μ represents the mean; σ represents the standard deviation; a represents the mean minus one standard deviation; b represents the mean plus one standard deviation; X represents the unprocessed observation; X norm represents the normalized data; X max Indicates the maximum value; X min Indicates the minimum value; Since the subsequent data concatenation process needs to be padded with 0, in order to distinguish it from the original data 0 which has practical significance, the Min-Max variant is used to normalize the data to between a and b; S205. Feature splicing processing: After normalization, the biomechanical data, musculoskeletal ultrasound data and clinical information data are filled with 0 using the surface electromyography data as the reference width, as shown in Table 1; the specific steps are as follows: The size of the biomechanical data is 8 12. The first six columns are abnormal sides, and the last six columns are normal sides. Six zeros are added to the end of the six abnormal side indicators, and six zeros are added to the end of the six normal side indicators. The shape after filling is 8. 24; musculoskeletal ultrasound data size is 1 20, the first ten columns are the abnormal side, and the last ten columns are the normal side. Add 2 zeros to the end of the first ten indicators (abnormal side) and add 2 zeros to the beginning of the last ten indicators (normal side). The shape after filling is 1 24; add 14 zeros to the end of the clinical information data, and the shape after filling is 1 24; The biomechanical data, surface electromyography data, musculoskeletal ultrasound data and clinical information data are spliced ​​in the 0th dimension to form 18 Matrix of 24; Table 1 Data feature concatenation results

[0033] S206. Rotation splicing: Rotate the feature to 18 The matrix of 24 is rotated and spliced ​​8 times to form 54 72 matrix, as shown in Table 2 and Figure 2-Figure 3 As shown. The original matrix is ​​retained at the center, and the other sub-matrices are truncated to varying degrees after rotation. Neither the rows nor the columns have time dimension information. After truncation, only local feature information is retained. There will be a lot of redundant information after splicing. In terms of model processing, batch normalization and L2 normal form regularization are required to avoid overfitting. Table 2 Matrix rotation splicing results

[0034] S207. Mirror processing: Flip the spliced ​​matrix horizontally and vertically, where the probability of horizontal flipping and vertical flipping is 50%; specifically: The first step is to flip the concatenated matrix horizontally along the vertical axis. There is a 50% chance of flipping horizontally and a 50% chance of remaining unchanged. The second step is to flip the result of the first step vertically along the horizontal axis. There is a 50% chance of vertical flipping and a 50% chance of keeping it unchanged. Since horizontal flipping and vertical flipping are independent of each other, the final result ratio is obtained by probability calculation: the probability of only horizontal flipping is 25%, the probability of only vertical flipping is 25%, the probability of both horizontal flipping and vertical flipping is 25%, and the probability of no flipping is 25%, as shown in Table 3; Table 3 Data mirroring processing combination probability

[0035] S208.Mixup data mixing: Randomly select two matrices from the matrix set processed by S207, mix the two matrices and labels by linear interpolation, and assign mixed weights to different categories. Given two input samples X i and X j and its corresponding label [Y i1 , Y i2 ] and [Y j1 , Y j2 ], new sample X generated by Mixup new and [Y new1 , Y new2 ], expressed as:

[0036] Among them [Y i1 , Y i2 ] represents the Ashworth grading and modified Tardieu scale rating level, and λ is the weight coefficient that conforms to the sampling in the Beta distribution, usually 0.4 or 0.6.

[0037] S3. Construct a deep learning model based on CNN network, multi-scale feature fusion and self-attention mechanism (see Figure 4 ); The deep learning model includes a first feature extraction module, a second feature extraction module, a third feature extraction module, a feature fusion module and a classification module in sequence; the first feature extraction module, the second feature extraction module, the third feature extraction module and the feature fusion module output a scale feature fx in sequence, x=1, 2, 3, 4, and finally the classification module outputs a specific category (the classification module has no features); Specifically, the first feature extraction module includes two-scale convolution and self-attention mechanisms to capture low-level features and global dependencies; the second feature extraction module includes two-scale convolution and self-attention mechanisms to capture mid-level features and global dependencies; the third feature extraction module includes a single-scale convolution and self-attention mechanism to capture high-level features and global dependencies; the feature fusion module includes an upsampling module, a jump connection module and a fusion convolution module to fuse low-level and high-level features; the classification module includes two fully connected classification layers for classification; The convolutions of the two scales are 3 3 convolution kernels of low-level convolution and 5 5 high-level convolution of convolution kernel; 3 3 The low-level convolution of the convolution kernel is filled with 1, and the extracted low-level features usually contain rich detail information; 5 The high-level convolution of the 5-convolution kernel is padded to 2, and the extracted high-level features usually contain more semantic information; The single-scale convolution is 3 3 low-level convolutions of convolution kernels; The calculation method of the self-attention mechanism is as follows: S301. Input feature vector X (size is n×d, where n is the sequence length and d is the feature dimension), and map it to query matrix Query (Q), key matrix Key (K) and value matrix Value (V) through three linear transformations (or fully connected layers): Q=XW Q K=XW K V=XW V Among them, W Q , W K and W V is a learnable weight matrix with dimensions d×d k (d k is the dimension of the key vector); Q, K, and V represent the query matrix, key matrix, and value matrix respectively; the dimensions of the calculated Q, K, and V are all n×d k , n is the sequence length, d is the feature dimension; S302. Calculate the attention score matrix: Its calculation is based on the dot product of the query matrix Q and the key matrix K transposed:

[0038] Among them, K T is the transpose of the key matrix K; QK T The dot product of the transpose of the query matrix Q and the key matrix K has a dimension of n×n. In order to prevent the dot product value from being too large and causing gradient instability, As a scaling factor, used to control the range of the dot product; S303. Use the Softmax function to normalize the scaled attention scores to obtain the attention weight matrix (Attention Weights):

[0039] The softmax function is calculated row by row to ensure that the sum of the weights of each row is 1, thereby converting the attention score matrix into a probability distribution, allowing the model to more clearly represent the relative importance of each eigenvalue to other eigenvalues:

[0040] Among them, A ij represents the attention score of the i-th eigenvalue to the j-th eigenvalue; exp(A ij ) is A ij The exponential function value, that is, e A ij ;∑ n k=1 exp(A ik ) is the sum of the exponential values ​​of all elements in the i-th row; S304. Use the attention weight matrix to perform weighted sum on the value matrix V to obtain the final output feature:

[0041] Among them, the dimension of the attention weight matrix (AttentionWeights) is n×n, and the dimension of the value matrix V is n×d k , the dimension of the output feature is n×d k .

[0042] The above deep learning model uses "parallel two-branch + serial skip connection structure" to perform multi-scale feature fusion. Among them, the low-level convolution uses 3 3 convolution kernels, padding is 1, the low-level features extracted usually contain rich details; while high-level convolution uses 5 The convolution kernel is 5 and the padding is 2. The extracted high-level features usually contain more semantic information. The padding ensures that the feature maps of each module have the same size and are fused along the first dimension, thereby improving the classification performance.

[0043] S4. Input the preprocessed data into the constructed deep learning model for training, optimize the model parameters through multiple rounds of iterations, and obtain the deep learning model for spasticity evaluation; the specific training strategy is as follows: Loss function: Two cross-entropy loss functions (CrossEntropyLoss) are used as objective functions. One cross-entropy loss function is used to calculate the loss between the assessment level of the modified Ashworth grading scale predicted by the model and the assessment level label of the true modified Ashworth grading scale, and the other cross-entropy loss function is used to calculate the loss between the assessment level of the modified Tardieu grading scale predicted by the model and the assessment level label of the true modified Tardieu grading scale.

[0044] Optimizer: Use the Adam optimizer, set the initial learning rate to 1e-3, the L2 norm regularization weight decay value to 5e-7, the exponential decay rate of the first-order moment estimate to 0.95, and the exponential decay rate of the second-order moment estimate to 0.999.

[0045] Evaluation indicators: Calculate accuracy, average precision, AUC and other indicators to evaluate model performance. Accuracy refers to the proportion of samples that the model predicts correctly, and is an important indicator for measuring the overall performance of the model. Average precision measures the accuracy of the model at different recall rates, and can more comprehensively evaluate the performance of the model at different classification thresholds, especially for data sets with imbalanced categories. AUC refers to the area under the ROC curve, which comprehensively evaluates the classification ability of the model at different thresholds, and can more comprehensively reflect the generalization ability of the model.

[0046] Model interpretability: Use the weight information of gradient-weighted class activation mapping and self-attention layer to explain the model's dependence on different modal data. Gradient-weighted class activation mapping is used to calculate the average gradient of each modality (biomechanical data, surface electromyography data, musculoskeletal ultrasound data, and clinical information data). The contribution of each modality to the model decision can be evaluated based on the average gradient score, thereby explaining the model's dependence on different modal data. The weight information of the self-attention layer (Self-attention-weights) is calculated. By analyzing the weight matrix, we can understand the model's attention to different features during the feature extraction process, thereby explaining how the model integrates different features for classification decisions.

[0047] S5. Input the data to be evaluated into the spasm evaluation deep learning model, and output the evaluation results of spasm and contracture.

[0048] The present invention also provides a spasm and contracture evaluation system based on multi-scale data feature fusion, comprising: a data acquisition module, a data preprocessing module, a model building module, a model training module and an evaluation module; A data acquisition module is used to collect biomechanical data, surface electromyography data, musculoskeletal ultrasound data, clinical information data, and the assessment levels of the modified Ashworth grading scale and the modified Tardieu grading scale of clinical patients with post-stroke spasticity and contracture, and send the data to a data preprocessing module; The data preprocessing module is used to preprocess the data. The preprocessing includes chain equation multiple filling, label encoding, normalization, feature splicing, data transformation and data enhancement. After preprocessing, the data is sent to the model training module. Specifically, data pairing is performed before normalization; data transformation includes rotation splicing and mirroring, and data enhancement is Mixup data mixing. Model building module, used to build a deep learning model based on CNN network, multi-scale feature fusion and self-attention mechanism; A model training module is used to input the preprocessed data into the constructed deep learning model for training, and optimize the model parameters through multiple rounds of iterations to obtain a deep learning model for spasticity evaluation; The evaluation module is used to input the data to be evaluated into the trained model and output the evaluation results of spasticity and contracture.

[0049] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0050] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for evaluating spasm and contracture based on multi-scale data feature fusion, characterized in that: The specific steps include: S1. Collect biomechanical data, surface electromyography data, musculoskeletal ultrasound data, and clinical information data of patients with post-stroke spasticity and contracture; record the assessment levels of the modified Ashworth grading scale and the modified Tardieu grading scale corresponding to the patients as label data; S2. Preprocessing the data collected in step S1, including chain equation multiple filling, label encoding, normalization, feature splicing, data transformation and data enhancement; S3. Constructing a deep learning model based on a CNN network, multi-scale feature fusion and self-attention mechanism; the deep learning model includes a first feature extraction module, a second feature extraction module, a third feature extraction module, a feature fusion module and a classification module in sequence; S4. Input the preprocessed data into the constructed deep learning model for training, optimize the model parameters through multiple rounds of iterations, and obtain a deep learning model for spasticity evaluation; S5. Input the data to be evaluated into the spasm evaluation deep learning model, and output the evaluation results of spasm and contracture.

2. The method for evaluating spasm and contracture based on multi-scale data feature fusion according to claim 1, characterized in that: In step S1, a stratified sampling method is used to collect data; The biomechanical data are indicators of the normal side and the abnormal side measured at 8 angular velocities, each side has 6 indicator items, and the shape of a single data item is 8 12; The surface electromyography data are indicators of the normal side and the abnormal side measured at 8 angular velocities. Each side has 12 indicator items, and the shape of a single data item is 8 twenty four; The musculoskeletal ultrasound data are divided into normal side and abnormal side, each side has 10 indicators, and the shape of a single data is 1 20; The clinical information data covers the basic information of the patient, with 10 indicators, and the shape of a single data item is 1 10; The modified Ashworth grading scale is used to assess the patient's muscle tension level. Different grades correspond to different degrees of increased muscle tension. The shape of a single data item is 1 1; The modified Tardieu grading scale is used to assess the degree of spasticity. Different grades correspond to different degrees of spasticity severity. The shape of a single data item is 1 1.

3. The method for evaluating spasm and contracture based on multi-scale data feature fusion according to claim 1, characterized in that: The data transformation in step S2 includes rotation splicing and mirroring, and data enhancement is Mixup data mixing; specifically, it includes the following sub-steps: S201. Chain equation multiple imputation: MICE multiple imputation method was used to construct MICE models based on biomechanical data, surface electromyography data, musculoskeletal ultrasound data and clinical information data, and the data with missing values ​​in some columns were imputed; S202. Label encoding: The modified Ashworth grading scale is mapped into 6 categories, where 0, 1, 1+, 2, 3, and 4 correspond to 0, 1, 2, 3, 4, and 5 respectively; the modified Tardieu grading scale is mapped into 6 categories, where 0, 1, 2, 3, 4, and 5 correspond to 0, 1, 2, 3, 4, and 5 respectively; S203. Data pairing: biomechanical data, surface electromyography data, musculoskeletal ultrasound data, clinical information data, and the assessment levels of the modified Ashworth grading scale and the modified Tardieu grading scale are paired and combined according to the patient's name to form a standardized data set; S204. Normalization processing: after pairing, the maximum and minimum values ​​of the biomechanical data, surface electromyography data, musculoskeletal ultrasound data, and clinical information data in the column direction are calculated, and the maximum and minimum values ​​are used for normalization in the row direction; S205. Feature splicing processing: After normalization processing, the biomechanical data, the musculoskeletal ultrasound data and the clinical information data are padded with 0 using the surface electromyography data as the reference width, and the biomechanical data, the surface electromyography data, the musculoskeletal ultrasound data and the clinical information data are spliced ​​in the 0th dimension to form a matrix; S206. Rotation splicing: Splice the matrix 8 times to form 54 Matrix of 72; S207. Mirror processing: first flip the spliced ​​matrix horizontally, and then flip it vertically; S208.Mixup data mixing: Randomly select two matrices from the matrix set processed by S207, mix the two matrices and labels by linear interpolation, and assign mixing weights to different categories.

4. The method for evaluating spasm and contracture based on multi-scale data feature fusion according to claim 3, characterized in that: The step S204 uses the Min-Max variant to normalize the data, which is expressed as: in μ represents the mean; σ represents the standard deviation; a represents the mean minus one standard deviation; b represents the mean plus one standard deviation; X represents the unprocessed observation; X norm represents the normalized data; X max Indicates the maximum value; X min Indicates the minimum value; The step S205 specifically includes: the biomechanical data size is 8 12. The first six columns are abnormal sides, and the last six columns are normal sides. Six zeros are added to the end of the six abnormal side indicators, and six zeros are added to the end of the six normal side indicators. The shape after filling is 8. 24; musculoskeletal ultrasound data size is 1 20, the first ten columns are abnormal side, the last ten columns are normal side, add 2 zeros after the ten abnormal side indicators, and add 2 zeros before the ten normal measurement indicators, the shape after filling is 1 24; add 14 zeros to the end of the clinical information data, and the shape after filling is 1 24; The biomechanical data, surface electromyography data, musculoskeletal ultrasound data and clinical information data are spliced ​​in the 0th dimension to form 18 Matrix of 24; In the matrix after rotation and splicing in step S206, the center position retains the original matrix, and other sub-matrices are truncated to varying degrees after rotation. After truncation, only local feature information is retained, and there is no time dimension information in rows and columns.

5. The method for evaluating spasm and contracture based on multi-scale data feature fusion according to claim 1, characterized in that: The first feature extraction module includes two scales of convolution and self-attention mechanism for capturing low-level features and global dependencies; the second feature extraction module includes two scales of convolution and self-attention mechanism for capturing mid-level features and global dependencies; the third feature extraction module includes single-scale convolution and self-attention mechanism for capturing high-level features and global dependencies; the feature fusion module includes an upsampling module, a jump connection module and a fusion convolution module for fusing low-level and high-level features; the classification module includes two fully connected classification layers for classification; the first feature extraction module, the second feature extraction module, the third feature extraction module and the feature fusion module output features fx of one scale in turn, x=1, 2, 3, 4; and finally the classification module outputs a specific category; The convolutions of the two scales are 3 3 convolution kernels of low-level convolution and 5 5 high-level convolution of convolution kernel; 3 3 The low-level convolution of the convolution kernel is filled with 1, and the extracted low-level features contain rich detail information; 5 The high-level convolution of the 5-convolution kernel is padded to 2, and the extracted high-level features contain more semantic information; The single-scale convolution is 3 3 convolution kernels for low-level convolution.

6. The method for evaluating spasm and contracture based on multi-scale data feature fusion according to claim 5, characterized in that: The calculation method of the self-attention mechanism is as follows: S301. Input feature vector X and map it to query matrix, key matrix and value matrix respectively through three linear transformations or fully connected layers: Q=XW Q K=XW K V=XW V Among them, W Q , W K and W V is a learnable weight matrix with dimensions d×d k , d k is the dimension of the key vector; Q, K, and V represent the query matrix, key matrix, and value matrix, respectively; the dimensions of the calculated query matrix, key matrix, and value matrix are all n×d k ; n is the sequence length, d is the feature dimension; S302. Calculate the attention score matrix based on the dot product of the query matrix Q and the key matrix K transpose: Among them, K T is the transpose of the key matrix K; QK T is the dot product of the transpose of the query matrix Q and the key matrix K, with dimension n×n; S303. Use the Softmax function to normalize the scaled attention scores to obtain the attention weight matrix: ; The softmax function is calculated row by row to ensure that the sum of the weights of each row is 1, thereby converting the attention score matrix into a probability distribution, indicating the relative importance of each eigenvalue to other eigenvalues: Among them, A ij represents the attention score of the i-th eigenvalue to the j-th eigenvalue; exp(A ij ) is A ij The exponential function value of n k=1 exp(A ik ) is the sum of the exponential values ​​of all elements in the i-th row; S304. Use the attention weight matrix to perform weighted sum on the value matrix V to obtain the final output feature: Among them, AttentionWeights represents the attention weight matrix, the dimension is n×n, and the dimension of the value matrix V is n×d k , the dimension of the output feature is n×d k .

7. The method for evaluating spasm and contracture based on multi-scale data feature fusion according to claim 1, characterized in that: The training strategy of step S4 is specifically as follows: The addition of two cross entropy loss functions is used as the objective function, one of which is used to calculate the loss between the modified Ashworth grading scale rating level predicted by the model and the true modified Ashworth grading scale rating level label, and the other cross entropy loss function is used to calculate the loss between the modified Tardieu grading scale rating level predicted by the model and the true modified Tardieu grading scale rating level label; the Adam optimizer is used, the initial learning rate is set to 1e-3, the L2 norm regularization weight decay value is 5e-7, the exponential decay rate of the first-order moment estimate is 0.95, and the exponential decay rate of the second-order moment estimate is 0.

999.

8. The method for evaluating spasm and contracture based on multi-scale data feature fusion according to claim 7, characterized in that: In the step S4 training process, gradient weighted class activation mapping is used to calculate the average gradient of biomechanical data, surface electromyography data, musculoskeletal ultrasound data and clinical information data, and the contribution of biomechanical data, surface electromyography data, musculoskeletal ultrasound data and clinical information data to model decision-making is evaluated according to the average gradient score, and the dependence of the model on surface electromyography data, musculoskeletal ultrasound data and clinical information data is explained; the evaluation indicators include accuracy, average precision and area under the ROC curve.

9. The method for evaluating spasm and contracture based on multi-scale data feature fusion according to claim 2, characterized in that: The biomechanical data include: maximum passive flexion angle of ankle joint, maximum passive extension angle of ankle joint, stretching moment, flexion moment, average power, and average work done; The specific indicators of the surface electromyography data include: medial gastrocnemius head-average electromyography value, lateral gastrocnemius head-average electromyography value, soleus muscle-average electromyography value, medial gastrocnemius head-root mean square value, lateral gastrocnemius head-root mean square value, soleus muscle-root mean square value, medial gastrocnemius head-median frequency, lateral gastrocnemius head-median frequency, soleus muscle-median frequency, medial gastrocnemius head-power frequency, lateral gastrocnemius head-power frequency, soleus muscle-power frequency; The specific indicators of the musculoskeletal ultrasound data include: fiber length, feather angle, medial gastrocnemius head - cross-sectional area, medial gastrocnemius head - muscle thickness, medial gastrocnemius head - shear wave velocity, lateral gastrocnemius head - cross-sectional area, lateral gastrocnemius head - muscle thickness, lateral gastrocnemius head - shear wave velocity, soleus muscle - muscle thickness, soleus muscle - shear wave velocity; Specific indicators of the clinical information data include: gender, age, height, weight, Barthel index, ankle passive joint range of motion, V1 speed limb reach angle, V3 speed stagnation angle, lower limb motor recovery stage, and lower limb motor function score.

10. A spasm and contracture evaluation system based on multi-scale data feature fusion, characterized in that: include: Data acquisition module, data preprocessing module, model building module, model training module and evaluation module; A data acquisition module is used to collect biomechanical data, surface electromyography data, musculoskeletal ultrasound data, clinical information data, and the assessment levels of the modified Ashworth grading scale and the modified Tardieu grading scale of clinical patients with post-stroke spasticity and contracture, and send the data to a data preprocessing module; The data preprocessing module is used to preprocess the data. The preprocessing includes chain equation multiple filling, label encoding, normalization, feature concatenation, data transformation and data enhancement. After preprocessing, the data is sent to the model training module. Model building module, used to build a deep learning model based on CNN network, multi-scale feature fusion and self-attention mechanism; A model training module is used to input the preprocessed data into the constructed deep learning model for training, and optimize the model parameters through multiple rounds of iterations to obtain a deep learning model for spasticity evaluation; The evaluation module is used to input the data to be evaluated into the trained model and output the evaluation results of spasticity and contracture.

Images