Motion function multi-dimensional intelligent evaluation method and system based on multi-modal hierarchical fusion network

Abstract

The invention discloses a motion function multi-dimensional intelligent evaluation method and system based on a multi-modal hierarchical fusion network. The method comprises the following steps: firstly, inputting multi-modal data, and receiving original data from a sensor; then data preprocessing and feature extraction are executed, and signal filtering, segmentation, alignment and feature extraction are completed; an MMHF-Net core model is adopted, and efficient fusion of multi-modal data is achieved; and finally, multi-scale evaluation result output is realized and is used for generating macroscopic, mesoscopic and microscopic three-level evaluation results. According to the method, kinematics, myoelectricity, electroencephalogram and visual information are deeply fused, the motion control process is comprehensively and accurately restored, and the evaluation precision is remarkably superior to that of a single-mode method. A multi-modal hierarchical fusion network MMHF-Net is provided, multi-modal space-time interaction can be modeled, key information can be dynamically focused, scale scores, sub-project indexes and recognition results are output, fine-grained clinical insight is provided, and excellent generalization performance is achieved based on multi-task learning and data enhancement.

Description

Technical Field

[0001] This invention belongs to the field of neurorehabilitation assessment technology, and in particular relates to a multi-dimensional intelligent assessment system and method for patients with limb motor dysfunction caused by neurological diseases such as stroke, spinal cord injury, and traumatic brain injury. Background Technology

[0002] With the increasing aging of the global population and the high incidence of neurological diseases such as stroke, traumatic brain injury, and spinal cord injury, limb motor dysfunction has become one of the major challenges in the field of rehabilitation medicine. Currently, standardized scales (such as the Fugl-Meyer Assessment, FMA; Wolf Motor Function Test, WMFT; Modified Ranking Scale, MRS; Barthel Index, BI, etc.) are widely used in clinical practice for subjective assessment of patients' motor function. While these scales are authoritative in rehabilitation medicine, the assessment process is highly dependent on the experience of rehabilitation physicians, and suffers from significant drawbacks such as high subjectivity, low efficiency, poor repeatability, and coarse granularity.

[0003] In recent years, with the development of wearable sensors, biomedical signal processing, and artificial intelligence technologies, researchers have begun to explore intelligent assessment methods based on sensor data and machine learning models. For example: motion analysis systems based on inertial measurement units (IMUs) collect patient kinematic data using accelerometers, gyroscopes, etc., extracting indicators such as joint angles, motion trajectories, and smoothness, and combining them with traditional machine learning models (such as SVM, KNN, BPNN) for score prediction. Muscle strength assessment systems based on surface electromyography (sEMG) signals assess muscle activation patterns and fatigue levels by analyzing muscle electrical activity characteristics (such as RMS, MPF, MF), with some studies using CNN or LSTM models for classification or regression prediction. Posture recognition systems based on visual sensors (such as Kinect, RealSense) extract key points of the human body using computer vision algorithms, analyzing the degree of action completion and coordination; some systems combine deep learning models for score prediction. Multimodal fusion assessment systems: In recent years, some studies have attempted to fuse IMU and sEMG, or IMU and visual data, using early or late fusion strategies to improve assessment accuracy. For example, some patents (such as CN114973062A) use the Transformer model for multimodal sentiment analysis, but their fusion strategy is a simple feature splicing, which fails to fully explore the deep correlation between modalities.

[0004] Although the aforementioned technologies have made some progress in promoting the objectivity and intelligence of rehabilitation assessment, the following significant shortcomings still exist:

[0005] The information dimensions are limited, leading to one-sided evaluation results. Most systems rely on only single-modal data (such as IMU or sEMG), which cannot fully reflect the complex physiological mechanisms of motor control. Motor function involves multiple links, including central nervous system commands (EEG), neural conduction (sEMG), skeletal execution (IMU), and environmental interaction (visual, force feedback). Single-modal data is insufficient to construct a complete functional profile.

[0006] The lack of clinical gold standard labels makes model outputs uninterpretable. Existing technologies are mostly based on small sample experimental data and lack labeled data aligned with clinical gold standard scales (such as FMA, MRS, and Barthel), making it difficult for clinicians to accept and trust the model output results. The models are mostly "black box" structures, only outputting a total score or classification result, and cannot provide interpretable information such as specific abnormal patterns or key feature contributions.

[0007] The simple model structure makes it difficult to model complex nonlinear relationships; traditional machine learning models (such as SVM and BPNN) rely on manual feature engineering, which is cumbersome and makes it difficult to capture high-dimensional, nonlinear, and time-dependent physiological signal features. Although a single CNN or LSTM model can process time-series data, it is insufficient in long sequence modeling, modality fusion, noise suppression, and has poor generalization performance.

[0008] Multimodal fusion strategies are crude and fail to uncover deep correlations between modalities. Existing fusion methods mostly involve early splicing or late voting, failing to consider the dynamic relationships and complementarities between different modalities. The lack of attention-based cross-modal information interaction results in insufficient discriminative power of fused features and limited evaluation accuracy.

[0009] The system lacks multi-dimensional, fine-grained assessment capabilities; most existing systems only predict a single total score (such as the FMA total score) and cannot simultaneously output fine-grained indicators such as muscle strength, coordination, and abnormal patterns. It cannot provide a "high-definition profile" of the patient's motor dysfunction, making it difficult to support the development of personalized rehabilitation plans and the tracking of treatment effectiveness.

[0010] The existing models exhibit poor adaptability and insufficient robustness to real-world clinical data. In actual clinical environments, sensor data is often affected by noise interference, wearing errors, and signal loss, leading to a significant performance degradation when data quality declines. The lack of robust designs to address issues such as missing data, missing modalities, and individual differences makes it difficult to operate stably in real-world clinical scenarios.

[0011] In summary, existing intelligent assessment technologies for limb motor dysfunction have significant shortcomings in terms of data dimensionality, label alignment, model structure, fusion strategies, assessment dimensions, and robustness, making it difficult to meet the actual clinical needs for objective, accurate, multi-dimensional, and interpretable assessments. Therefore, there is an urgent need for an intelligent assessment system that integrates multimodal data, uses clinical gold standards for supervision, is based on advanced deep learning models, and possesses multi-task output and robustness enhancement mechanisms to achieve comprehensive, quantitative, and intelligent assessment of motor dysfunction. Summary of the Invention

[0012] The purpose of this invention is to solve the problems mentioned in the background art by proposing an intelligent assessment system and method for limb motor dysfunction. By collecting multimodal data of patients and using deep learning algorithms to establish a comprehensive assessment model, the invention achieves multidimensional intelligent quantitative assessment of patients with limb motor dysfunction, thereby improving the accuracy, objectivity and efficiency of the assessment.

[0013] To achieve the objective of this invention, a multi-dimensional intelligent evaluation method for motion function based on a multimodal hierarchical fusion network is disclosed, comprising the following steps:

[0014] Step 1: Multimodal data input, receiving raw data from wearable inertial sensor IMU data, surface electromyography (sEMG) data, electroencephalography (EEG) data, and visual sensor;

[0015] Step 2: Data preprocessing and feature extraction, completing signal filtering, segmentation, alignment, and feature extraction;

[0016] Step 3: The MMHF-Net core model is adopted, which integrates a multimodal encoder, a cross-modal dynamic attention mechanism (CMDA), and a hierarchical fusion module to achieve efficient fusion of multimodal data.

[0017] Step 4: Output multi-scale assessment results to generate three levels of assessment results: macroscopic, mesoscopic (sub-item score), and microscopic (fine-grained motion index), forming a complete "high-definition portrait" of motor function.

[0018] Furthermore, in step 1, multimodal data is collected from the patient while performing assessment tasks, which are specifically divided into three categories: upper limb tasks, lower limb tasks, and whole-body functional tasks. Wearable inertial measurement unit (IMU) data records the acceleration, angular velocity, and orientation information of the limbs, reflecting the movement trajectory and dynamic characteristics. Surface electromyography (sEMG) data records muscle activity potentials, reflecting muscle activation patterns and coordination characteristics. Electroencephalography (EEG) data records cerebral cortex activity, reflecting the neurophysiological state of motor control. Visual sensor data records the movement process through a camera, providing a macroscopic view of the movement performance. The data acquisition from all sensors is precisely synchronized through a central control unit.

[0019] Further, in step 2, the acquired raw signal is cleaned, denoised, segmented, and its features are extracted, converting it into a format suitable for deep learning model processing. Specific steps include:

[0020] Step 2-1: Signal denoising and filtering; appropriate digital filtering techniques are used to address the characteristics of different signal modes.

[0021] Step 2-2, Data Segmentation and Alignment: Based on the start and end time markers of the evaluation task, the continuous long-term signal is segmented into short-term data segments corresponding to a single task.

[0022] Steps 2-3: Feature extraction and normalization; For each data segment, extract features from the original signal that reflect the state of motion function.

[0023] Steps 2-4: Data quality assessment; Design a quality assessment network to assess the data quality of each modality.

[0024] Further, in step 2-1, the IMU data processing first applies a mid-range filter to remove outliers and spike noise, then uses a Butterworth low-pass filter (cutoff frequency 20 Hz, order 4) to remove high-frequency noise and muscle tremor interference. For angular velocity and acceleration signals, a Kalman filter is applied for signal smoothing and fusion to improve the accuracy of motion trajectory estimation. The sEMG signal processing first applies a bandpass filter (20-450 Hz, order 4) to remove motion artifacts and power frequency interference, then uses a notch filter (50 / 60 Hz, bandwidth 1 Hz) to remove power supply noise, followed by an adaptive Wiener filter to remove possible crosstalk noise, and finally performs envelope extraction to obtain the time-varying curve of muscle activity intensity. The EEG signal processing first applies a bandpass filter (0.5-45 Hz, order 6) to retain the main EEG frequency bands, then uses a notch filter to remove 50 / 60 Hz noise. Hz power supply noise is then eliminated. Independent component analysis (ICA) algorithm is used to identify and remove EOG and EMG artifacts. Finally, spatial filtering is applied to further improve signal quality. For video data processing, Gaussian filtering is used to smooth the image and reduce the impact of noise. Background segmentation algorithm is used to extract the foreground human contour and temporal median filtering is applied to reduce inter-frame jitter.

[0025] In step 2-2, the start / end markers of the tasks in the data acquisition system are used to accurately extract the data segments corresponding to each task. For data without clear markers, an automatic segmentation algorithm based on energy threshold and motion pattern recognition is used to identify the start and end times of the tasks. Data with different sampling rates are resampled to a unified standard (e.g., 100 Hz) using interpolation algorithms (such as cubic spline interpolation) to ensure that the data of all modes are strictly aligned on the time axis. Task data of different lengths are standardized into fixed-length sequences using dynamic time warping (DTW) or linear interpolation methods to facilitate batch training.

[0026] In steps 2-3, IMU feature extraction includes time-domain features (mean, variance, peak value, zero-crossing rate, range), frequency-domain features (dominant frequency, frequency band energy distribution), statistical features (quartiles, entropy), and kinematic features (joint angles, angular velocity, acceleration, smoothness, jitter); sEMG feature extraction includes time-domain features (RMS, average rectified value, peak value), frequency-domain features (median frequency, average frequency, power spectrum), time-frequency features (wavelet coefficients), and muscle synergy features (intermuscular correlation, activation delay, co-activation ratio); EEG feature extraction includes frequency band energy (δ, θ, α, β). The extraction of visual features includes skeletal keypoint coordinates, joint angles, trajectory features (velocity, acceleration, curvature), posture descriptors (HOG features), and temporal motion features (optical flow field features). All extracted features are normalized using Z-score (subtract the mean and divide by the standard deviation) or Min-Max normalized (scaled to the [0,1] interval) to eliminate dimensional differences between different sensors and individuals and accelerate model convergence.

[0027] In steps 2-4, a quality assessment network is designed to evaluate the quality of data for each modality:

[0028]

[0029] in, X m The input to the QualityNet quality assessment network represents the raw data or preprocessed feature data from the m-th modality (IMU / sEMG / EEG / vision). ∈[0,1] represents the quality score of the modality, which will be used for subsequent adaptive weight allocation; the quality assessment network is trained through supervised learning, using expert-annotated signal quality ratings as labels, and learns to identify noise levels, signal stability, and sensor detachment issues; in its specific implementation, QualityNet adopts a lightweight CNN structure, containing 3 convolutional layers and 2 fully connected layers, and outputs a single quality score; among them: IMU quality assessment focuses on signal stability, motion coherence, and sensor drift; sEMG quality assessment focuses on signal-to-noise ratio, baseline stability, and electrode contact quality; EEG quality assessment focuses on artifact severity, impedance stability, and signal integrity; visual quality assessment focuses on image sharpness, occlusion severity, and keypoint detection confidence.

[0030] Through the above preprocessing steps, the original multimodal sensor data is transformed into standardized and aligned feature representations, providing high-quality input for subsequent deep learning models. At the same time, it preserves the key temporal patterns and multidimensional features in the original signals, ensuring that the model can fully learn the comprehensive representation of the patient's motor function state.

[0031] Furthermore, in step 3, the MMHF-Net core model processes heterogeneous data through a multi-level fusion strategy to achieve high-precision rehabilitation assessment. The MMHF-Net model includes a multimodal specific encoder (IMU, sEMG, EEG, vision), a cross-modal dynamic attention mechanism (CMDA), a hierarchical multi-level fusion module (early, middle, and late fusion), an adaptive personalized assessment module, and a multi-task learning and joint optimization module. The workflow of step 3 is as follows: receiving the preprocessed multimodal feature sequence, and learning a deep feature vector that can comprehensively represent the patient's motor function status through the feature extraction, fusion, and encoding network within the model. This model can not only handle the heterogeneity of different modal data, but also effectively capture the complex spatiotemporal dependencies between them. The model design fully considers the characteristics of low signal-to-noise ratio and large individual differences in biomedical signals, and ensures stable performance in real clinical environments by introducing robustness mechanisms. Finally, this module outputs a deep feature vector that integrates multimodal information, providing a foundation for the generation of subsequent multi-dimensional assessment results.

[0032] Furthermore, multimodal specific encoders include inertial sensor encoders (IMU Encoders), surface electromyography (sEMG) encoders, electroencephalography (EEG) encoders, and visual encoders.

[0033] The inertial sensor encoder (IMU Encoder) employs an improved temporal convolutional network (TCN), combining a dual-scale structure of LocalTCN (small receptive field for capturing fine-grained features) and GlobalTCN (large receptive field for capturing large-scale patterns). It also incorporates a motion pattern attention mechanism to automatically emphasize clinically significant motion patterns and introduces phase-aware processing and spatiotemporal resolution adaptive adjustment to improve the accuracy of motion trajectory feature extraction.

[0034]

[0035]

[0036]

[0037] in, It is a fine-grained inertial sensor feature extracted by a local temporal convolutional network, superscript Representing local features, LocalTCN uses a smaller receptive field to capture fine-grained motion features. It is a large-scale inertial sensor pattern feature extracted by a global temporal convolutional network, superscript Represents global features; GlobalTCN uses a larger receptive field to capture large-scale motion patterns; The input is the raw data from the inertial sensor; This is the final inertial sensor feature representation after fusing local and global features; Concat represents the feature concatenation operation. This refers to the attention mechanism in motion patterns.

[0038] TCN is defined as:

[0039]

[0040] in: It is the first The expansion factor of the layer For layer normalization operation, To correct the activation function of the linear unit, Conv1D() is a one-dimensional convolution operation. This is a temporal convolutional network of layer l;

[0041] The motion pattern attention mechanism is defined as:

[0042]

[0043] in: For the sigmoid function, For the learnable weight matrix and bias vector of the motion pattern attention mechanism, This is a global average pooling operation;

[0044] The surface electromyography (sEMG) encoder employs a CNN-GRU hybrid architecture, along with a frequency domain analysis branch (based on FFT transformation) and a muscle co-analysis module (combining nonnegative matrix factorization and deep learning). This enables multi-domain fusion of features from the time domain, frequency domain, and nonlinear dynamic domain. Simultaneously, it integrates dynamic fatigue detection and multi-muscle interaction modeling to comprehensively characterize electromyographic activity.

[0045]

[0046]

[0047]

[0048]

[0049] ,

[0050] in, The surface electromyography time-domain features are extracted for the time-domain branch. The surface electromyography frequency domain features are extracted from the frequency domain branch. These are muscle coordination features extracted by the muscle coordination analysis module. To represent the final surface electromyography features after fusing time-domain, frequency-domain, and collaborative features, The input is the raw surface electromyography data. This is a convolutional neural network specifically for surface electromyography (SEMG), where GRU(⋅) is a gated recurrent unit network. Indicates Fourier transform, This indicates that it is specifically designed for analyzing muscle synergistic activation pattern networks. Represents a feature fusion network;

[0051] The EEG Encoder is based on an improved Transformer architecture, incorporating a spatial awareness attention mechanism (combined with spatial masking of electrode physical distance) to capture spatial associations in brain regions. It simultaneously introduces a functional connectivity network (extracting functional connectivity features between brain regions through graph neural networks) and designs dedicated processing channels for different EEG frequency bands. It also integrates a motor imagery classifier to assess differences in motor control at the neural level.

[0052]

[0053]

[0054]

[0055] in, Spatial features of EEG extracted for spatial perception and attention mechanisms; the superscript spatial indicates space. The EEG functional connectivity features extracted from the functional connectivity network are represented by the superscript conn, which indicates connectivity. This represents the final EEG feature representation after fusing spatial and connectivity features; The input is the raw EEG data; SpatialTransformer(⋅) is the spatial perception Transformer; FunctionalConnectivityNet(⋅) is the functional connectivity network; α is the fusion weight coefficient, with a value range of [0,1], used to balance the contribution ratio of spatial features and connectivity features; (1−α) is the complement of α, representing the weight of connectivity features;

[0056] in, Defined as:

[0057]

[0058]

[0059]

[0060] in, The input features are defined by LN(⋅); LN(⋅) represents layer normalization; MultiHead(⋅) represents multi-head attention; Concat represents feature concatenation. For the first The output of each attention head; The total number of attention heads; The output projection matrix is ​​used; Attention(⋅) is the attention mechanism function. For X and Matrix multiplication; for and Matrix multiplication; for and Matrix multiplication;

[0061] The spatial awareness attention mechanism is defined as:

[0062]

[0063] in, For queries, i.e., the Query matrix; The key is the key matrix; The value matrix is ​​the matrix of values. for and transpose matrix Matrix multiplication, the superscript T denotes matrix transpose; is the dimension of the key vector; softmax(⋅) is the normalization exponential function; It is a spatial mask based on the physical distance of brain electrodes, defined as:

[0064]

[0065] in, It is an electrode and The Euclidean distance between them It is a scaling parameter;

[0066] Functional connectivity networks are defined as follows:

[0067]

[0068] in, The input is EEG data; ConnectivityMatrix(⋅) is the function to calculate the functional connectivity matrix between brain regions; GNN is a graph neural network used to extract brain network features.

[0069] The Visual Encoder adopts a 3D-ResNet structure and combines a motion focusing module based on optical flow estimation to focus on regions with significant motion changes. It also integrates prior attention based on human anatomy, multi-scale temporal modeling, and pose estimation with motion analysis to improve the ability to recognize complex motion patterns.

[0070]

[0071] The motion focusing module is defined as follows:

[0072]

[0073] .

[0074] in, It represents the visual features output by the visual encoder; 3D-ResNetVisual is a three-dimensional residual network specifically for vision. The input is the raw visual data, i.e., the video frame sequence; MotionFocus(⋅) is the motion focusing module; Let represent the video frame at time t; The video frame at time t-1 is the previous frame; Conv3D(⋅) is a 3D convolution operation used to extract motion features; This is the visual representation after fusing motion features; Concat represents the feature concatenation operation; MotionFocus(X) represents calculating the motion focus features of adjacent frames for the input sequence X.

[0075] Furthermore, the cross-modal dynamic attention mechanism CMDA dynamically adjusts weights based on the quality scores of each modality. It constructs bidirectional information interaction between modalities through multi-head cross-modal mutual attention, designs an inter-layer attention weight transfer mechanism to guide high-level feature extraction from low-level attention patterns, and introduces contrastive learning to promote semantic representation alignment between modalities to address the modal heterogeneity problem, ensuring that the model can maintain stable performance even in noisy or signal loss scenarios.

[0076]

[0077] in, , It comes from modality eigenvectors, It is modal Quality assessment score; These are learnable projection matrices used to generate queries, keys, and values, respectively. It is the dimension of the projection space; It is modal For modes Attention weight matrix; It is modal From modality via attention mechanism Enhanced feature representation; It is modal The adaptive interaction strength coefficient between them; It is modal Enhanced features resulting from cross-modal information interaction.

[0078] The hierarchical multi-level fusion module implements differentiated fusion strategies at different network levels: early feature fusion performs convolutional fusion on low-level features of each modality in the early stage of extraction to achieve basic information integration; mid-stage feature fusion constructs inter-modal interaction associations through attention mechanisms to enhance feature discrimination; late-stage feature fusion calculates adaptive weights based on quality and task relevance at the decision layer to achieve weighted fusion.

[0079] Early feature fusion (low-level) involves basic fusion at the initial stage of feature extraction:

[0080]

[0081] in, The features extracted for each modality in the first layer (low level) This represents the joint features after early fusion.

[0082] Intermediate Feature Fusion (Intermediate Level): Interactive fusion is performed during the intermediate stage of feature extraction.

[0083]

[0084]

[0085] in, These are the learnable parameters for the gating mechanism; For modality Gating weights; Indicates the mode The set of all modal features outside of, Represents element-wise multiplication;

[0086] Late-stage feature fusion (advanced) involves weighted fusion at the decision level:

[0087]

[0088]

[0089] in, For the prediction output based on the features of a single mode i, The weights are adaptively calculated based on quality and task relevance. and The learnable parameters of the weight calculation network are used to calculate the weights. It is modal The quality assessment score, The final prediction output after multimodal fusion is presented; Neural Architecture Search (NAS) is introduced to automatically select the optimal fusion path, enabling the model to dynamically optimize the fusion strategy according to task requirements and data characteristics; Multi-granularity fusion design captures full-spectrum features from low-level signal characteristics to high-level semantic information, and improves the model's expressive power and generalization performance through progressive information distillation and multimodal mutual learning mechanisms.

[0090]

[0091]

[0092] Among them, NASNet is a neural architecture search network used to learn the optimal fusion strategy; The optimal fusion strategy obtained through neural architecture search; These are the optimal fusion features selected after NASNet optimization.

[0093] Furthermore, the adaptive personalized assessment module achieves accurate adaptation of the model to individual differences based on the meta-learning framework. It adopts an improved Reptile algorithm to optimize meta-parameters and introduces personalized adaptation parameters at multiple layers of the network, breaking through the limitations of the traditional single adaptation layer. At the same time, it integrates the patient's static characteristics (age, disease course, injury type) and dynamic status, and realizes personalized adjustment of network features through the feature linear modulation FiLM mechanism. It introduces an external memory mechanism to store the representation and adaptation strategies of similar patient cases, accelerating the personalization process for new patients. It designs a progressive parameter update strategy to avoid catastrophic forgetting. Even when patient data is limited, it can quickly adapt through parameter fine-tuning to ensure the accuracy of assessment.

[0094] The adaptive personalized assessment module employs a Reptile-based meta-learning algorithm, which is more suitable for the distribution of multimodal data.

[0095]

[0096]

[0097] in: These are meta-parameters; they are temporary parameters updated after the inner loop. The learning rate for the inner loop; It is the loss on the support set; The learning rate for the outer loop;

[0098] The adaptive personalized assessment module integrates the subject's static characteristics and dynamic state:

[0099]

[0100]

[0101]

[0102] For subjects with limited data:

[0103]

[0104] in, This represents the patient's static feature embedding (age, gender, disease duration, injury type, etc.); This indicates the patient's dynamic state embedding (current physiological state, fatigue level, etc.); This is the fused patient feature vector; This represents the final feature representation after hierarchical fusion. These are personalized features modulated based on patient characteristics; is the feature scaling factor, generated from patient feature P via tanh activation; This is the feature bias term, generated by a linear transformation of patient features P; These are the learnable parameters of the FiLM mechanism; These are personalized parameters fine-tuned for specific patients. It is a loss function based on limited patient data.

[0105] Furthermore, the multi-task learning and joint optimization module achieves joint prediction of three levels of assessment indicators: macro, meso, and micro, through a hierarchical prediction architecture. At the macro level, it outputs the total scores of clinical scales such as FMA, WMFT, MRS, and Barthel Index. At the meso level, it predicts the scores of sub-items of each scale. At the micro level, it quantifies fine-grained indicators such as joint mobility, muscle coordination, motion smoothness, muscle activation patterns, and abnormal movement pattern recognition. Moreover, high-level prediction guides low-level prediction, enhancing assessment consistency. To improve the medical rationality and accuracy of the prediction, a constraint-enhanced loss function is designed, including prediction loss for calculating the prediction error of each indicator, scale correlation loss using scale correlation constraints, medical constraint loss incorporating medical prior knowledge (such as FMA improvement taking precedence over Barthel score improvement), and assessment consistency loss to ensure the consistency of predictions across different modalities. At the same time, it introduces feature sparsity, modal complementarity, and time-series consistency regularization strategies, and optimizes the weights of each loss component through the validation set to form an overall loss function to guide model training.

[0106] The hierarchical prediction architecture of the multi-task learning and joint optimization module is as follows:

[0107]

[0108] in, This represents a macroeconomic assessment and forecast. This represents the predicted value from the mid-level assessment. This represents the predicted value based on micro-level assessment. These correspond to the prediction functions for the three levels of assessment.

[0109] Design a loss function that incorporates medical knowledge, and predict the loss as follows: For each evaluation metric, calculate the prediction error.

[0110]

[0111] in, It is an indicator Weighting coefficients; Let the mean squared error loss function be used. To predict losses;

[0112] The scale correlation loss is: utilizing the correlation between different scales

[0113]

[0114] in, yes The correlation coefficient between them; It is a scale and The weighting coefficients of the correlation between them; This represents the loss of scale relevance.

[0115] The medical prior knowledge constraint loss is:

[0116] Using prior medical knowledge as a constraint

[0117]

[0118] in: It is a single-task medical constraint; These are constraints on the relationships between tasks; These are the weighting coefficients for the constraints on the relationships between tasks; Loss due to constraints imposed by prior medical knowledge;

[0119] For example, if we know that improvement in the FMA score should precede improvement in the Barthel score:

[0120]

[0121] in It is a constant determined from medical knowledge.

[0122] The consistency loss is assessed as follows:

[0123] Ensure consistency between predictions derived from different modalities:

[0124]

[0125] in: From modality and The exported prediction results It is modal and Weighting coefficients for consistency between them; To assess the loss of consistency;

[0126] Overall loss function design regularization strategy:

[0127]

[0128] in, It is a feature sparsity regularization, using the L1 norm; It is a modal diversity regularization that uses negative KL divergence to promote modal complementarity; It is time-series consistency regularization, which ensures the stability of features at adjacent time points; For the network The weight matrix of the layer; KL(⋅∣∣⋅) is the Kullback-Leibler divergence, which measures the difference between two distributions; These are the characteristic representations of times t and t−1; This is the regularization loss;

[0129] Overall losses:

[0130]

[0131] in These are the weights of different loss components, determined by performance on the validation set; This is the overall loss function.

[0132] To achieve the objectives of this invention, a multi-dimensional intelligent assessment system for motor function based on a multimodal hierarchical fusion network is also disclosed. This system includes a multimodal data input layer for receiving raw data from wearable inertial measurement units (IMUs), surface electromyography (sEMG) data, electroencephalography (EEG) data, and visual sensors; a data preprocessing and feature extraction module for performing signal filtering, segmentation, alignment, and feature extraction; an MMHF-Net core model integrating a multimodal encoder, a cross-modal dynamic attention mechanism (CMDA), and a hierarchical fusion module for efficient fusion of multimodal data; and a multi-scale assessment result output layer for generating macroscopic (total scale score), mesoscopic (sub-item score), and microscopic (fine-grained motor indicators) three-level assessment results, forming a complete "high-definition portrait" of motor function.

[0133] Compared with existing technologies, the significant advancements of this invention are: 1) Comprehensive and accurate assessment: For the first time, it deeply integrates kinematic, electromyographic, electroencephalographic, and visual information to comprehensively reconstruct the motor control process, and the assessment accuracy far exceeds that of single-modal methods; 2) Strong model innovation: The proposed multimodal hierarchical fusion network (MMHF-Net) model can effectively model the complex spatiotemporal interaction relationships between multimodal data and dynamically focus on key information; 3) Rich output dimensions: It simultaneously outputs scale scores, sub-item indicators, and pattern recognition results, providing multi-dimensional and fine-grained clinical insights; 4) Superior generalization performance: The multi-task learning framework and data augmentation strategy ensure that the model can still maintain strong generalization ability under limited data.

[0134] To more clearly illustrate the functional characteristics and structural parameters of the present invention, further explanation is provided below in conjunction with the accompanying drawings and specific embodiments. Attached Figure Description

[0135] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0136] Figure 1 This is the overall system architecture diagram;

[0137] Figure 2 This is a detailed structural diagram of the core model MMHF-Net;

[0138] Figure 3 This is a schematic diagram of the cross-modal dynamic attention mechanism (CMDA).

[0139] Figure 4 It is a multi-layered fusion framework structure diagram;

[0140] Figure 5 It is a flowchart of the evaluation methods and system application. Detailed Implementation

[0141] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0142] The purpose of this invention is to achieve multi-dimensional intelligent quantitative assessment of patients with limb motor dysfunction by collecting multimodal data from patients and using deep learning algorithms to establish a comprehensive assessment model, thereby improving the accuracy, objectivity, and efficiency of the assessment. Specific objectives include:

[0143] 1. A multimodal data-driven intelligent assessment method and system is provided, which integrates data from multiple sensors such as inertial measurement unit (IMU), surface electromyography (sEMG), electroencephalography (EEG), visual sensors and force feedback devices to comprehensively characterize the patient's motor function status from multiple dimensions such as kinematics, neuromuscular control, central command and environmental interaction.

[0144] 2. Construct a supervised learning framework centered on the clinical gold standard, and use clinical scale scores such as FMA (Fugl-Meyer Assessment), WMFT (Wolf Motor Function Test), MRS (Modified Rankin Scale), and Barthel Index as labels for model training to ensure that the model output results are highly aligned with clinical assessment standards and improve the clinical interpretability and credibility of the results.

[0145] 3. Design a multimodal hierarchical fusion deep learning model architecture to jointly predict multiple clinical scale scores and fine-grained motor function indicators (such as joint range of motion, muscle synergistic contraction, abnormal movement patterns, etc.) within a unified model framework, so as to achieve multi-dimensional assessment from macroscopic functional state to microscopic motor characteristics.

[0146] 4. A multimodal fusion strategy based on cross-modal attention mechanism is proposed. Through the cross-modal dynamic attention (CMDA) module, information interaction and dynamic weighting between heterogeneous modalities are realized, which solves the problem that traditional fusion methods cannot explore deep correlations between modalities.

[0147] 5. Develop an adaptive personalized assessment framework that can automatically adjust model parameters based on the patient's individual characteristics (such as age, condition, degree of injury, etc.), improve the model's adaptability to individual differences, and achieve accurate assessment.

[0148] The overall system architecture is as follows Figure 1 As shown, the overall architecture of this system consists of four main parts: a multimodal data input layer, which is responsible for receiving raw data from IMU, sEMG, EEG and visual sensors; a data preprocessing and feature extraction module, which completes signal filtering, segmentation, alignment and feature extraction; the MMHF-Net core model, which integrates a multimodal encoder, cross-modal dynamic attention mechanism (CMDA) and hierarchical fusion module to achieve efficient fusion of multimodal data; and a multi-scale evaluation result output layer, which generates three levels of evaluation results: macro (total scale score), meso (sub-item score) and micro (fine-grained motion index), forming a complete "high-definition portrait" of motor function.

[0149] Figure 1 The overall architecture of an intelligent assessment system for limb motor dysfunction based on a multimodal hierarchical fusion network is demonstrated, comprising four main parts: a multimodal data input layer, responsible for receiving raw data from IMU, sEMG, EEG, and visual sensors; a data preprocessing and feature extraction module, which completes signal filtering, segmentation, alignment, and feature extraction; the MMHF-Net core model, which integrates a multimodal encoder, a cross-modal dynamic attention mechanism (CMDA), and a hierarchical fusion module to achieve efficient fusion of multimodal data; and a multi-scale assessment result output layer, which generates three levels of assessment results: macro (total scale score), meso (sub-item score), and micro (fine-grained motor indicators), forming a complete "high-definition portrait" of motor function.

[0150] Figure 2This diagram details the internal structure of the MMHF-Net model, including multimodal specific encoders (IMU, sEMG, EEG, vision), cross-modal dynamic attention mechanism (CMDA), hierarchical multi-level fusion modules (early, mid, and late fusion), and multi-task learning and joint optimization modules. The diagram clearly reveals how the model achieves efficient fusion of multimodal data through multi-level, cross-modal interactions, and provides structured support for the "macro-meta-micro" three-level evaluation, enhancing the model's interpretability.

[0151] Figure 3 This paper demonstrates a cross-modal dynamic attention mechanism (CMDA) that dynamically adjusts the weights of each modality based on their quality scores, achieving bidirectional information interaction between modalities through a multi-head attention mechanism. This mechanism includes a quality assessment module, cross-modal feature enhancement, and adaptive interaction intensity learning, effectively improving the model's robustness to noise interference or signal loss.

[0152] Figure 4 The diagram illustrates the three-level fusion strategy employed in MMHF-Net: early feature fusion performs convolutional fusion at the low-level feature level; mid-level feature fusion enhances intermodal interactions through an attention mechanism; and late-level feature fusion performs weighted fusion based on quality and task relevance at the decision layer. The diagram also demonstrates a dynamic fusion path selection mechanism implemented through Neural Architecture Search (NASNet) to optimize the fusion strategy.

[0153] Figure 5 This diagram showcases the complete workflow of the evaluation system, from multimodal data acquisition, preprocessing, and feature extraction to multi-level fusion and personalized adaptation of the MMHF-Net model, ultimately outputting multi-dimensional evaluation results. The diagram clearly illustrates the system's application path in real-world clinical scenarios, highlighting its closed-loop processing capabilities from data to decision.

[0154] The specific implementation process includes:

[0155] (a) Multimodal Data Acquisition: Responsible for collecting multimodal data of patients while performing assessment tasks, including:

[0156] 1. Wearable inertial measurement unit (IMU) data: records the acceleration, angular velocity and orientation information of the limbs, reflecting the motion trajectory and dynamic characteristics;

[0157] 2. Surface electromyography (sEMG) data: Records muscle activity potentials, reflecting muscle activation patterns and synergistic characteristics;

[0158] 3. Electroencephalogram (EEG) data: Records the activity of the cerebral cortex, reflecting the neurophysiological state of motor control;

[0159] 4. Visual sensor data: Records the motion process through cameras, providing a macroscopic view of the motion performance;

[0160] Data acquisition from all sensors will be precisely synchronized through a central control unit. The specific solution is as follows:

[0161] IMU Sensors: Multiple IMU units (with a sampling rate of 200 Hz and a range of ±16 g acceleration and ±2000 dps angular velocity) are worn on the patient's torso, upper arms, forearms, back of hands, thighs, calves, and dorsum of feet via elastic straps or specialized fixation devices. Each IMU unit contains a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer, providing a total of 9 degrees of freedom, enabling precise capture of limb movement in three-dimensional space.

[0162] sEMG sensor: Depending on the evaluation task, multi-channel sEMG electrode pads (sampling rate of 1000 Hz, bandwidth of 20-450 Hz) are attached to the surface of the target muscle group, such as the biceps, triceps, deltoid, quadriceps, and tibialis anterior. A dual-electrode differential amplification configuration is used, with two electrodes for each muscle to record the difference in electromyographic signals, effectively suppressing common-mode noise and improving the signal-to-noise ratio.

[0163] EEG sensor: Adopting the international 10-20 system standard, high-density EEG electrode sensors (sampling rate of 1000 Hz, bandwidth of 0.1-100 Hz) are worn on the subject to ensure good contact between the electrodes and the scalp. The focus is on brain regions related to motor control, such as the motor cortex, premotor cortex, and supplementary motor area, recording changes in brain electrical activity.

[0164] Visual sensors: One or more depth cameras (such as Intel RealSense) are placed in front of and to the sides of the assessment area to ensure complete capture of the patient's full-body movements. High-precision human pose estimation is achieved by combining depth information with RGB images.

[0165] Synchronization: Data streams from all sensors are synchronized via the Lab Streaming Layer (LSL) protocol or hardware trigger signals to ensure precise temporal alignment and uniform sampling rates across different modalities. The system employs the NTP protocol for timestamp calibration, ensuring cross-device synchronization errors are less than 1ms. The master acquisition device sends a synchronization pulse signal, which other devices receive as a reference time point. Interpolation algorithms are used to achieve time alignment of data from multiple devices.

[0166] Simultaneously, clinical gold standard scores (FMA, WMFT, MRS, Barthel Index, etc.) are recorded as labels for supervised learning. Scores are independently completed by at least two experienced rehabilitation physicians, and their average is used as the final label to ensure label reliability. If the difference between the scores of two physicians exceeds a preset threshold (e.g., 10%), a third physician is brought in for additional evaluation, and the final score is reached through consensus.

[0167] (II) Assessment Action Design: The assessment task design is based on the logic of clinical rehabilitation assessment, closely integrating the assessment dimensions of internationally recognized scales (such as FMA and Barthel Index) and the needs of Activities of Daily Living (ADL), constructing a three-level task system of "upper limb - lower limb - whole body function" to cover key scenarios for assessing motor dysfunction. Standardized operating procedures are established for all tasks, clearly defining the starting posture, movement execution process, and ending criteria. Rehabilitation physicians provide unified guidance and demonstrations to patients before the assessment. Each task requires patients to repeat it 3-5 times, and the 3 records with the best data stability are used for analysis to ensure the reliability and repeatability of the assessment results.

[0168] The assessment tasks are specifically divided into three categories: upper limb tasks, lower limb tasks, and whole-body functional tasks. The difficulty of the tasks gradually increases from basic joint movements to complex daily activities, which can accurately identify subtle abnormalities in patients with mild functional impairments and effectively assess the basic motor abilities of patients with severe impairments. At the same time, by simulating daily life scenarios, the assessment results are ensured to have predictive value for patients' actual living abilities.

[0169] Upper limb assessment tasks mainly include isolated and compound movements of the shoulder, elbow, wrist, and finger joints in different planes, such as: shoulder flexion, abduction, and rotation; elbow flexion and extension; forearm pronation and supination; wrist flexion, extension, and lateralization; and fine motor skills such as grasping, finger opposition, and extension of the fingers. Specific tasks include:

[0170] Shoulder flexion: The patient sits with their arm hanging down and raises it forward to the maximum range of motion, holds for 2 seconds, and then returns to the starting position.

[0171] Shoulder abduction: The patient sits with their arm hanging down naturally. The patient raises their upper arm to the side to the maximum range of motion, holds for 2 seconds, and then returns to the starting position.

[0172] Elbow flexion and extension: The patient sits down and starts from a fully extended elbow position, flexing the elbow to the maximum range, holding for 2 seconds, and then returning to the starting position.

[0173] Forearm rotation: The patient sits with the elbow flexed at 90 degrees. Starting from a neutral position, perform maximum pronation and supination movements respectively.

[0174] Wrist flexion and extension: The patient sits with their forearm supported on a table. Starting from a neutral position, perform maximum dorsiflexion and palmar flexion exercises respectively.

[0175] Grasping action: The patient sits, reaches out to grab a cup on the table, lifts it up, and puts it back in its original position.

[0176] Fingertip to finger: The patient sits down and touches the tips of the other four fingers with the thumb in turn, then returns to the starting position.

[0177] Lower limb assessment tasks focus on functional movement and postural control of the hip, knee, and ankle joints, such as hip and knee flexion and extension, ankle dorsiflexion and plantarflexion, as well as support and mobility tasks such as transitions from sitting to standing, stepping in place, and short-distance walking. Specific tasks include:

[0178] Hip flexion: The patient lies supine and begins in a fully extended position, flexing the hip joint to the maximum range, holding for 2 seconds, and then returning to the starting position.

[0179] Knee flexion and extension: The patient sits with the knee fully extended, flexes the knee to the maximum range, holds for 2 seconds, and then returns to the starting position.

[0180] Ankle dorsiflexion and plantarflexion: The patient sits in a neutral position and performs maximum dorsiflexion and plantarflexion exercises respectively.

[0181] Sitting-standing transfer: The patient stands up from a standard height chair, holds the position for 3 seconds, and then sits back down.

[0182] Stepping in place: The patient stands and steps in place with knees raised for 30 seconds, maintaining balance.

[0183] 10-meter walk: The patient walks 10 meters in a straight line on a flat surface, and the time and gait characteristics are recorded.

[0184] The whole-body function assessment focuses on balance and daily activity performance, and consists of two core tasks. The first is a static balance test, designed based on items I-II of the FMA balance assessment; the second is a daily activity simulation (dressing) task, referencing the Barthel Index "dressing" item.

[0185] Each task clearly defines the starting posture, movement specifications, completion standards, and precautions, and is guided on-site by rehabilitation physicians using standardized instructions and demonstrations to ensure the standardization and comparability of data collection. This task system not only comprehensively and progressively challenges patients' motor function limits, thereby exposing their functional impairment patterns, but also its movement structure highly corresponds to clinical scale items, laying a solid foundation for establishing a precise mapping between multimodal data and clinical scores in the model.

[0186] (III) Data Preprocessing: Before entering the core evaluation model, the collected raw multimodal data must undergo a series of rigorous preprocessing and feature engineering steps. The main function of this module is to clean, denoise, segment, and extract features from the raw signals, converting them into a format suitable for deep learning model processing. The specific process is as follows:

[0187] 1. Signal denoising and filtering: Appropriate digital filtering techniques are used to address the characteristics of signals with different modes.

[0188] IMU data processing: First, a medium-range filter is applied to remove outliers and spike noise. Then, a Butterworth low-pass filter (cutoff frequency 20 Hz, order 4) is used to remove high-frequency noise and muscle tremor interference. For angular velocity and acceleration signals, a Kalman filter is applied for signal smoothing and fusion to improve the accuracy of motion trajectory estimation.

[0189] sEMG signal processing: First, a bandpass filter (20-450Hz, order 4) is applied to remove motion artifacts and power frequency interference. Then, a notch filter (50 / 60 Hz, bandwidth 1 Hz) is used to remove power supply noise. Next, an adaptive Wiener filter is applied to remove possible crosstalk noise. Finally, envelope extraction is performed to obtain the time-varying curve of muscle activity intensity.

[0190] EEG signal processing: Bandpass filtering (0.5-45 Hz, order 6) was applied to preserve the main EEG frequency bands, followed by notch filtering to remove 50 / 60 Hz power supply noise. Independent component analysis (ICA) was then used to identify and remove artifacts such as electrooculography (EOG) and electromyography (EMG), and finally, spatial filtering was applied to further improve signal quality.

[0191] Video data processing: Gaussian filtering is applied to smooth the image and reduce the impact of noise. A background segmentation algorithm is used to extract the foreground human body contour, and temporal median filtering is applied to reduce inter-frame jitter.

[0192] 2. Data Segmentation and Alignment: Based on the start and end time markers of the evaluation task, the continuous long-term signal is segmented into short-term data segments corresponding to a single task.

[0193] Event-based tagging: Utilizing the start / end markers of tasks in the data acquisition system, accurately extract the data segments corresponding to each task.

[0194] Automatic segmentation algorithm: For data without clear labels, an automatic segmentation algorithm based on energy threshold and motion pattern recognition is used to identify the start and end times of the task.

[0195] Time alignment: Data with different sampling rates are resampled to a unified standard (e.g., 100 Hz) using interpolation algorithms (e.g., cubic spline interpolation) to ensure that data from all modalities are strictly aligned on the time axis.

[0196] Length standardization: Task data of different lengths are standardized into fixed-length sequences through dynamic time warping (DTW) or linear interpolation methods to facilitate batch training.

[0197] 3. Feature extraction and normalization: For each data segment, features that reflect the state of motion function are extracted from the original signal.

[0198] IMU feature extraction includes time-domain features (mean, variance, peak value, zero-crossing rate, range), frequency-domain features (dominant frequency, frequency band energy distribution), statistical features (quartiles, entropy), and kinematic features (joint angles, angular velocity, acceleration, smoothness, jitter).

[0199] sEMG feature extraction includes time-domain features (RMS, average rectified value, peak value), frequency-domain features (median frequency, average frequency, power spectrum), time-frequency features (wavelet coefficients), and muscle synergy features (intermuscular correlation, activation delay, co-activation ratio).

[0200] EEG feature extraction includes band energy (δ, θ, α, β, γ band power), time-frequency features (wavelet coefficients), complexity features (sample entropy, Lempel-Ziv complexity), and functional connectivity features (coherence, phase-locked value, migration entropy).

[0201] Visual feature extraction includes skeletal keypoint coordinates, joint angles, trajectory features (velocity, acceleration, curvature), pose descriptors (HOG features), and temporal motion features (optical flow field features).

[0202] Feature normalization: All extracted features are normalized using Z-score (subtract the mean and divide by the standard deviation) or Min-Max normalization (scaled to the [0,1] interval) to eliminate dimensional differences between different sensors and individuals and accelerate model convergence.

[0203] 4. Data Quality Assessment: Design a quality assessment network to evaluate the data quality of each modality.

[0204]

[0205] in, ∈[0,1] represents the quality score of the mode, which will be used for subsequent adaptive weight allocation. The quality assessment network is trained through supervised learning, using expert-annotated signal quality ratings as labels, and learns to identify problems such as noise level, signal stability, and sensor detachment.

[0206] In its implementation, QualityNet employs a lightweight CNN architecture, consisting of three convolutional layers and two fully connected layers, outputting a single quality score. Specifically:

[0207] IMU quality assessment focuses on signal stability, motion coherence, and sensor drift.

[0208] sEMG quality assessment focuses on signal-to-noise ratio, baseline stability, and electrode contact quality;

[0209] EEG quality assessment focuses on the degree of artifacts, impedance stability, and signal integrity;

[0210] Visual quality assessment focuses on image sharpness, occlusion level, and keypoint detection confidence.

[0211] Through the above preprocessing steps, the original multimodal sensor data is transformed into standardized and aligned feature representations, providing high-quality input for subsequent deep learning models. At the same time, it preserves the key temporal patterns and multidimensional features in the original signals, ensuring that the model can fully learn the comprehensive representation of the patient's motor function state.

[0212] (iv) Multimodal Hierarchical Fusion Network (MMHF-Net) Model Construction and Training. The MMHF-Net architecture design is based on the following key theoretical framework, which supports its effectiveness in multimodal medical signal processing:

[0213] Information Bottleneck Theory: MMHF-Net follows the principle that optimal feature representation should simultaneously maximize mutual information with the target variable and minimize redundancy. This theory supports our use of a hierarchical fusion method to progressively extract key information across modalities, reducing noise and redundancy. Compared to traditional simple concatenation methods, the design based on information bottleneck theory can significantly improve information extraction efficiency and noise resistance.

[0214] Multimodal learning theory: Our design addresses three fundamental challenges in the field of multimodal learning: (i) the heterogeneity gap between modalities, (ii) the confusion between shared and modality-specific information, and (iii) robustness to missing modalities. A three-level fusion strategy specifically addresses these challenges by bridging the semantic gap between different modalities through learning joint representations at multiple levels of abstraction.

[0215] Neuroscience-inspired architectural design: The network structure simulates the hierarchical information processing observed in the human nervous system, from spinal reflexes to cortical planning, with sensorimotor integration occurring at multiple levels. This biomimicry approach is highly consistent with clinical understanding of motor function, making the model more interpretable and clinically acceptable.

[0216] Attention Mechanism Foundation: Our cross-modal dynamic attention mechanism is built on the theoretical work of cognitive science on selective attention and information filtering, enabling the network to focus computational resources on the most relevant signals and features, significantly improving processing efficiency and accuracy.

[0217] Meta-learning framework: Personalized components are based on the theoretical framework of meta-learning, particularly optimization-based meta-learning, which provides mathematical guarantees for rapid adaptation to new patients under limited data conditions. Compared with traditional transfer learning methods, the meta-learning framework can better handle high inter-individual variability.

[0218] MMHF-Net is the core innovation of this invention, comprising an innovative deep learning model specifically designed for multimodal motor function assessment. This network processes heterogeneous data through a multi-level fusion strategy to achieve high-precision rehabilitation assessment. Its basic workflow is as follows: receiving multimodal feature sequences from a preprocessing module, and learning a deep feature vector that comprehensively represents the patient's motor function status through its internal feature extraction, fusion, and encoding networks. This model not only handles the heterogeneity of different modal data but also effectively captures the complex spatiotemporal dependencies between them. The model design fully considers the characteristics of low signal-to-noise ratio and large individual differences in biomedical signals, and introduces robustness mechanisms to ensure stable performance in real clinical environments. Finally, this module outputs a high-dimensional, information-rich feature representation, providing a foundation for the subsequent generation of multi-dimensional assessment results.

[0219] The multimodal hierarchical fusion network is the core innovation of this invention. The detailed structure of the MMHF-Net model is as follows: Figure 2 As shown, the model includes multimodal specific encoders (IMU, sEMG, EEG, vision), a cross-modal dynamic attention mechanism (CMDA), hierarchical multi-level fusion modules (early, mid, and late-stage fusion), and a multi-task learning and joint optimization module. This figure clearly reveals how the model achieves efficient fusion of multimodal data through multi-level, cross-modal interactions, and provides structured support for the "macro-meta-micro" three-level evaluation, enhancing the model's interpretability.

[0220] The network architecture includes the following main components:

[0221] 1. Multimodal specific encoder

[0222] (1) Inertial sensor encoder (IMU Encoder)

[0223] The IMU encoder employs an improved temporal convolutional network (TCN), combining a dual-scale structure of LocalTCN (small receptive field capturing fine-grained features) and GlobalTCN (large receptive field capturing large-scale patterns). It also incorporates a motion pattern attention mechanism to automatically emphasize clinically significant motion patterns, while introducing phase-aware processing and adaptive adjustment of spatiotemporal resolution to improve the accuracy of motion trajectory feature extraction.

[0224]

[0225]

[0226]

[0227] in, It is a fine-grained inertial sensor feature extracted by a local temporal convolutional network, superscript Representing local features, LocalTCN uses a smaller receptive field to capture fine-grained motion features. It is a large-scale inertial sensor pattern feature extracted by a global temporal convolutional network, superscript Represents global features; GlobalTCN uses a larger receptive field to capture large-scale motion patterns; The input is the raw data from the inertial sensor; This is the final inertial sensor feature representation after fusing local and global features; Concat represents the feature concatenation operation. This refers to the attention mechanism in motion patterns.

[0228] TCN is defined as:

[0229]

[0230] in: It is the first The expansion factor of the layer For layer normalization operation, To correct the activation function of the linear unit, Conv1D() is a one-dimensional convolution operation. This is the l-th layer of the temporal convolutional network.

[0231] The motion pattern attention mechanism is defined as:

[0232]

[0233] in: For the sigmoid function, For the learnable weight matrix and bias vector of the motion pattern attention mechanism, This is a global average pooling operation.

[0234] (2) Surface electromyography encoder (sEMG Encoder)

[0235] The sEMG encoder employs a CNN-GRU hybrid architecture, along with a frequency domain analysis branch (based on FFT transformation) and a muscle co-analysis module (combining nonnegative matrix factorization and deep learning), to achieve multi-domain fusion of features from the time domain, frequency domain, and nonlinear dynamic domain. It also integrates dynamic fatigue detection and multi-muscle interaction modeling to comprehensively characterize electromyographic activity.

[0236]

[0237]

[0238]

[0239]

[0240]

[0241] in, The surface electromyography time-domain features are extracted for the time-domain branch. The surface electromyography frequency domain features are extracted from the frequency domain branch. These are muscle coordination features extracted by the muscle coordination analysis module. To represent the final surface electromyography features after fusing time-domain, frequency-domain, and collaborative features, The input is the raw surface electromyography data. This is a convolutional neural network specifically for surface electromyography (SEMG), where GRU(⋅) is a gated recurrent unit network. Indicates Fourier transform, This indicates that it is specifically designed for analyzing muscle synergistic activation pattern networks. This represents a feature fusion network.

[0242] (3) EEG Encoder

[0243] The EEG encoder is based on an improved Transformer architecture, incorporating a spatial awareness attention mechanism (combined with spatial masking of electrode physical distance) to capture spatial associations in brain regions. It simultaneously introduces a functional connectivity network (extracting functional connectivity features between brain regions through graph neural networks) and designs dedicated processing channels for different EEG frequency bands. It also integrates a motor imagery classifier to assess differences in motor control at the neural level.

[0244]

[0245]

[0246]

[0247] in, Spatial features of EEG extracted for spatial perception and attention mechanisms; the superscript spatial indicates space. The EEG functional connectivity features extracted from the functional connectivity network are represented by the superscript conn, which indicates connectivity. This represents the final EEG feature representation after fusing spatial and connectivity features; The input is the raw EEG data; SpatialTransformer(⋅) is the spatial perception Transformer; FunctionalConnectivityNet(⋅) is the functional connectivity network; α is the fusion weight coefficient, with a value range of [0,1], used to balance the contribution ratio of spatial features and connectivity features; (1−α) is the complement of α, representing the weight of connectivity features;

[0248] in, Defined as:

[0249]

[0250]

[0251]

[0252] in, The input features are defined by LN(⋅); LN(⋅) represents layer normalization; MultiHead(⋅) represents multi-head attention; Concat represents feature concatenation. For the first The output of each attention head; The total number of attention heads; The output projection matrix is ​​used; Attention(⋅) is the attention mechanism function. For X and Matrix multiplication; for and Matrix multiplication; for and Matrix multiplication;

[0253] The spatial awareness attention mechanism is defined as:

[0254]

[0255] in, For queries, i.e., the Query matrix; The key is the key matrix; The value matrix is ​​the matrix of values. for and transpose matrix Matrix multiplication, the superscript T denotes matrix transpose; is the dimension of the key vector; softmax(⋅) is the normalization exponential function; It is a spatial mask based on the physical distance of brain electrodes, defined as:

[0256]

[0257] in, It is an electrode and The Euclidean distance between them It is a scaling parameter.

[0258] Functional connectivity networks are defined as follows:

[0259]

[0260] in, The input is EEG data; ConnectivityMatrix(⋅) is the function to calculate the functional connectivity matrix between brain regions; GNN is a graph neural network used to extract brain network features.

[0261] (4) Visual Encoder

[0262] The visual encoder employs a 3D-ResNet structure, combined with a motion-focusing module based on optical flow estimation, to concentrate on regions with significant motion changes. It also integrates prior attention based on human anatomy, multi-scale temporal modeling, and pose estimation with motion analysis to enhance the ability to recognize complex motion patterns.

[0263]

[0264] The motion focusing module is defined as follows:

[0265]

[0266]

[0267] in, It represents the visual features output by the visual encoder; 3D-ResNetVisual is a three-dimensional residual network specifically for vision. The input is the raw visual data, i.e., the video frame sequence; MotionFocus(⋅) is the motion focusing module; Let represent the video frame at time t; The video frame at time t-1 is the previous frame; Conv3D(⋅) is a 3D convolution operation used to extract motion features; This is the visual representation after fusing motion features; Concat represents the feature concatenation operation; MotionFocus(X) represents calculating the motion focus features of adjacent frames for the input sequence X.

[0268] 2. Cross-modal Dynamic Attention Mechanism

[0269] Cross-modal dynamic attention (CMDA) is key to achieving efficient multimodal fusion. It dynamically adjusts weights based on the quality scores of each modality, constructs bidirectional information exchange between modalities through multi-head cross-modal mutual attention, designs an inter-layer attention weight transfer mechanism to guide high-level feature extraction from low-level attention patterns, and introduces contrastive learning to promote semantic representation alignment between modalities. This effectively addresses the modal heterogeneity problem and ensures stable performance of the model even in noisy or signal-loss scenarios.

[0270]

[0271] in, , It comes from modality eigenvectors, It is modal Quality assessment score; These are learnable projection matrices used to generate queries, keys, and values, respectively. It is the dimension of the projection space; It is modal For modes Attention weight matrix; It is modal From modality via attention mechanism Enhanced feature representation; It is modal The adaptive interaction strength coefficient between them; It is modal Enhanced features resulting from cross-modal information interaction.

[0272] like Figure 3 As shown, the Cross-Modal Dynamic Attention (CMDA) mechanism dynamically adjusts the weights of each modality based on their quality scores, achieving bidirectional information interaction between modalities through a multi-head attention mechanism. This mechanism includes a quality assessment module, cross-modal feature enhancement, and adaptive interaction intensity learning, effectively improving the model's robustness to noise interference or signal loss.

[0273] 3. Hierarchical Multi-level Fusion Framework

[0274] The hierarchical multi-level fusion framework implements differentiated fusion strategies at different network levels: early feature fusion performs convolutional fusion on low-level features of each modality in the early stage of extraction to achieve basic information integration; mid-stage feature fusion constructs inter-modal interaction associations through attention mechanisms to enhance feature discrimination; late-stage feature fusion calculates adaptive weights based on quality and task relevance at the decision layer to achieve weighted fusion.

[0275] Multi-layered fusion framework structure such as Figure 4 As shown, the figure illustrates the three-level fusion strategy employed in MMHF-Net: early feature fusion performs convolutional fusion at the low-level feature level; mid-level feature fusion enhances intermodal interactions through an attention mechanism; and late-level feature fusion performs weighted fusion based on quality and task relevance at the decision layer. The figure also demonstrates a dynamic fusion path selection mechanism implemented through Neural Architecture Search (NASNet) to optimize the fusion strategy.

[0276] (1) Early feature fusion (low level)

[0277] Basic fusion is performed in the early stages of feature extraction:

[0278]

[0279] in, The features extracted for each modality in the first layer (low level) This represents the joint features after early fusion.

[0280] (2) Mid-term feature fusion (intermediate level)

[0281] Interactive fusion is performed during the mid-stage of feature extraction:

[0282]

[0283]

[0284] in, These are the learnable parameters for the gating mechanism; For modality Gating weights; Indicates the mode The set of all modal features outside of, This represents element-wise multiplication.

[0285] (3) Late-stage feature fusion (advanced)

[0286] Weighted fusion at the decision-making level:

[0287]

[0288]

[0289]

[0290] in, For the prediction output based on the single-modal i feature, The weights are adaptively calculated based on quality and task relevance. and The learnable parameters of the weight calculation network are used to calculate the weights. It is modal The quality assessment score, This is the final prediction output after multimodal fusion.

[0291] (4) Dynamic fusion path selection

[0292] By introducing Neural Architecture Search (NAS) to automatically select the optimal fusion path, the model can dynamically optimize the fusion strategy based on task requirements and data characteristics. This multi-granularity fusion design can capture full-spectrum features from low-level signal characteristics to high-level semantic information. Through progressive information distillation and multimodal mutual learning mechanisms, it significantly improves the model's expressive power and generalization performance.

[0293]

[0294]

[0295] Among them, NASNet is a neural architecture search network used to learn the optimal fusion strategy; The optimal fusion strategy obtained through neural architecture search; These are the optimal fusion features selected after NASNet optimization.

[0296] 4. Adaptive Personalized Assessment Module

[0297] The adaptive personalized assessment module achieves accurate adaptation of the model to individual differences based on the meta-learning framework. It adopts an improved Reptile algorithm to optimize meta-parameters and introduces personalized adaptation parameters at multiple layers of the network, breaking through the limitations of traditional single adaptation layers. At the same time, it integrates the patient's static characteristics (age, disease course, lesion type) and dynamic state, and realizes personalized adjustment of network features through the Feature Linear Modulation (FiLM) mechanism. It introduces an external memory mechanism to store the representation and adaptation strategies of similar patient cases, accelerating the personalization process for new patients. It designs a progressive parameter update strategy to avoid catastrophic forgetting. Even with limited patient data, it can quickly adapt through parameter fine-tuning to ensure the accuracy of assessment.

[0298] (1) Improved meta-learning framework

[0299] Employing a Reptile-based meta-learning algorithm is more suitable for multimodal data distributions.

[0300]

[0301]

[0302] in: These are meta-parameters; they are temporary parameters updated after the inner loop. ; It is the loss on the support set; The learning rate for the outer loop;

[0303] (2) Conditioning of individual characteristics of subjects

[0304] Integrating the static characteristics and dynamic states of the subjects:

[0305]

[0306]

[0307]

[0308] For subjects with limited data:

[0309]

[0310] in, This represents the patient's static feature embedding (age, gender, disease duration, injury type, etc.); This indicates the patient's dynamic state embedding (current physiological state, fatigue level, etc.); This is the fused patient feature vector; This represents the final feature representation after hierarchical fusion. These are personalized features modulated based on patient characteristics; is the feature scaling factor, generated from patient feature P via tanh activation; This is the feature bias term, generated by a linear transformation of patient features P; These are the learnable parameters of the FiLM mechanism; These are personalized parameters fine-tuned for specific patients. It is a loss function based on limited patient data.

[0311] 5. Multi-task Learning and Joint Optimization Framework

[0312] The multi-task learning and joint optimization module achieves joint prediction of three levels of evaluation indicators: macro, meso, and micro, through a hierarchical prediction architecture. At the macro level, it outputs the total score of the FMA, WMFT, MRS, and Barthel index scales. At the meso level, it predicts the scores of each scale sub-item. At the micro level, it quantifies fine-grained indicators such as joint mobility, muscle coordination, motion smoothness, and muscle synergistic contraction patterns. Furthermore, high-level prediction guides low-level prediction, enhancing evaluation consistency. To improve the medical rationality and accuracy of the prediction, a constraint-enhanced loss function is designed, including prediction loss for calculating the prediction error of each indicator, scale correlation loss using scale correlation constraints, medical constraint loss incorporating medical prior knowledge, and evaluation consistency loss to ensure the consistency of predictions across different modalities. At the same time, it introduces feature sparsity, modal complementarity, and time-series consistency regularization strategies. The weights of each loss component are optimized through the validation set to form an overall loss function that guides model training.

[0313] (1) Hierarchical multi-task prediction network

[0314] Design a hierarchical prediction architecture:

[0315]

[0316] in: This represents a macroeconomic assessment and forecast. This represents the predicted value from the mid-level assessment. This represents the predicted value based on micro-level assessment. These correspond to the prediction functions / networks for the three levels of evaluation.

[0317] (2) Constraint-enhanced loss function

[0318] Design a loss function that incorporates medical knowledge:

[0319] ①Predicting losses:

[0320] For each evaluation metric, calculate the prediction error:

[0321]

[0322] in, It is an indicator Weighting coefficients; Let the mean squared error loss function be used. To predict losses.

[0323] ②Scale correlation loss:

[0324] Utilizing the correlation between different scales:

[0325]

[0326] in, yes The correlation coefficient between them; It is a scale and The weighting coefficients of the correlation between them; This represents the loss of scale relevance.

[0327] ③ Loss of constraint from prior medical knowledge:

[0328] Using prior medical knowledge as a constraint:

[0329]

[0330] in: It is a single-task medical constraint; These are constraints on the relationships between tasks; These are the weighting coefficients for the constraints on the relationships between tasks; Loss due to constraints imposed by prior medical knowledge.

[0331] For example, if we know that improvement in the FMA score should precede improvement in the Barthel score:

[0332]

[0333] in It is a constant determined from medical knowledge.

[0334] ④ Assess the loss of consistency:

[0335] Ensure consistency between predictions derived from different modalities:

[0336]

[0337] in: From modality and The exported prediction results It is modal and Weighting coefficients for consistency between them; To assess the loss of consistency.

[0338] (3) Overall loss function

[0339] Design a regularization strategy:

[0340]

[0341] in, It is a feature sparsity regularization, using the L1 norm; It is a modal diversity regularization that uses negative KL divergence to promote modal complementarity; It is time-series consistency regularization, which ensures the stability of features at adjacent time points; For the network The weight matrix of the layer; KL(⋅∣∣⋅) is the Kullback-Leibler divergence, which measures the difference between two distributions; These are the characteristic representations of times t and t−1; This is the regularization loss.

[0342] Overall losses:

[0343]

[0344] in These are the weights of different loss components, determined by performance on the validation set; This is the overall loss function.

[0345] (v) Model training and optimization strategies

[0346] To ensure model performance, the system employs scientific training and optimization strategies:

[0347] 1. Hyperparameter settings:

[0348] Optimizer: AdamW (weight decay 1e-4), initial learning rate 1e-3, cosine annealing scheduling (T_max=50, eta_min=1e-6).

[0349] Batch size: 32, training epochs: 100, early stopping strategy (patience=10, evaluation metric is weighted average loss on the validation set).

[0350] Regularization: Dropout rate 0.3 (fully connected layers), Batch Normalization (between CNN and LSTM), L2 regularization (Transformer attention weights);

[0351] Gradient clipping: To prevent gradient explosion, the maximum gradient norm is set to 1.0;

[0352] Learning rate warm-up: The first 5 epochs use a linear learning rate for warm-up, which helps the model to be initialized stably;

[0353] Mixed precision training: FP16 mixed precision training is used to accelerate the calculation process and save GPU memory.

[0354] 2. Details of Meta-Learning Training:

[0355] The support set / query set partition ratio is 1:4, and the meta-learning rate α=0.01, β=0.001;

[0356] Patient condition information is incorporated into the model through an embedding layer (64 dimensions) and then concatenated with fused features to participate in personalized adaptation.

[0357] The meta-learning inner loop updates the step count by 5 steps, and the outer loop updates the step count by 1000 steps;

[0358] A hybrid strategy combining MAML (Model-Agnostic Meta-Learning) and Reptile is adopted, combining the advantages of both.

[0359] Task sampling strategy: During the meta-training phase, tasks are sampled according to the inverse probability of the patient score distribution to ensure that rare functional states are fully learned.

[0360] 3. Validation and evaluation methods:

[0361] Five-fold cross-validation was used to evaluate the model performance to ensure robust and reliable results.

[0362] The Leave-One-Patient-Out validation strategy was used to assess the performance of individualized adaptation;

[0363] Performance metrics include: mean squared error (MSE), Pearson correlation coefficient, consistency correlation coefficient (ICC), and mean absolute error (MAE).

[0364] Through the above training and optimization strategies, the MMHF-Net model can fully learn the deep features and intermodal correlations in multimodal data, and quickly adapt to the individual characteristics of new patients, achieving high-precision, multi-dimensional assessment of motor dysfunction.

[0365] The innovative aspects of this invention are as follows:

[0366] 1. Innovative Multimodal Hierarchical Fusion Network (MMHF-Net): This paper proposes a multi-level fusion strategy, including early feature fusion (low-level), mid-level interaction fusion (intermediate-level), and late-level decision fusion (high-level). It also introduces a cross-modal dynamic attention mechanism (CMDA) to dynamically adjust modal weights based on signal quality, enhancing the fusion effect. This network can effectively capture the complex spatiotemporal interactions between multimodal data, significantly improving the efficiency and evaluation accuracy of heterogeneous data fusion.

[0367] 2. Deep Fusion of Clinical Multimodal Data: This invention integrates data from multiple sensors, including inertial measurement units (IMU), surface electromyography (sEMG), electroencephalography (EEG), and visual sensors, to comprehensively assess motor function from multiple dimensions such as kinematics, neuromuscular control, central commands, and environmental interaction. A supervised learning framework is constructed around the clinical gold standard, using scores from scales such as FMA, WMFT, MRS, and Barthel Index as labels to ensure that the model output aligns with clinical standards. Unlike the simple data splicing in existing technologies, this invention employs a multi-level interactive fusion strategy to establish deep semantic connections between modalities, effectively addressing the problems of modal heterogeneity and information inconsistency.

[0368] 3. Adaptive Assessment and Joint Optimization Framework: Based on meta-learning (Reptile algorithm), this framework integrates static and dynamic patient features to achieve personalized model adaptation and enhance adaptability to individual differences. An enhanced loss function is designed, incorporating multiple constraints such as prior medical knowledge, scale relevance, and prediction consistency, to improve the model's medical rationality and generalization ability. This invention breaks through the limitations of traditional single adaptation layers by introducing personalized adaptation parameters at multiple network layers, combined with conditional modulation based on physiological characteristics, to achieve more refined individualized assessment.

[0369] 4. Innovative Multi-Dimensional Assessment Capabilities: Breaking away from the limitations of existing systems that only output a single total score, we construct a three-tiered assessment system: macro-, meso-, and micro-level. At the macro level, we output clinical scale scores; at the meso level, we cover the performance of each scale's sub-items; and at the micro level, we precisely quantify fine-grained indicators such as joint range of motion, motion smoothness, and muscle activation patterns, forming a complete "high-definition portrait" of motor dysfunction. This provides data support for personalized rehabilitation program development and efficacy tracking. Through hierarchical predictive dependency modeling, high-level predictions can guide low-level predictions, improving the consistency and accuracy of multi-dimensional assessments.

[0370] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0371] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for intelligent evaluation of motor function in multiple dimensions based on a multi-modal hierarchical fusion network, characterized in that, Comprising the following steps: Step 1, multi-modal data input, receiving raw data from wearable inertial sensor IMU data, surface electromyography sEMG data, electroencephalogram EEG data and visual sensor; Step 2, data preprocessing and feature extraction, completing signal filtering, segmentation, alignment and feature extraction; Step 3, using MMHF-Net core model, the model integrates multi-modal encoder, cross-modal dynamic attention mechanism CMDA and hierarchical fusion module, realizing efficient fusion of multi-modal data; Step 4, multi-scale evaluation result output, used to generate macro, meso and micro three-level evaluation results.

2. The method of claim 1, wherein, In step 1, multi-modal data of patients during the execution of evaluation tasks is collected, and the evaluation tasks are specifically divided into three categories: upper limb tasks, lower limb tasks and whole body function tasks; wearable inertial sensor IMU data records the acceleration, angular velocity and orientation information of the limb, reflecting the motion trajectory and dynamic characteristics; surface electromyography sEMG data records the muscle activity potential, reflecting the muscle activation mode and coordination characteristics; electroencephalogram EEG data records the activity of the cerebral cortex, reflecting the neurophysiological state of motor control; The visual sensor data records the movement process through the camera, providing a macroscopic perspective of the movement performance; The data collection of all sensors is accurately synchronized through a central control unit.

3. The method of claim 1, wherein the method is characterized by: In step 2, the collected raw signals are cleaned, denoised, segmented and feature extracted, and converted into a format suitable for deep learning model processing, the specific steps include: Step 2-1, signal denoising and filtering; according to the signal characteristics of different modalities, corresponding digital filtering techniques are used; Step 2-2, data segmentation and alignment; according to the start and end time markers of the evaluation task, the continuous long-time signal is segmented into short-term data segments corresponding to a single task; Step 2-3, feature extraction and normalization; for each data segment, features reflecting the motor function state are extracted from the original signal; Step 2-4, data quality evaluation; a quality evaluation network is designed to evaluate the quality of each modality data.

4. The multi-modal hierarchical fusion network-based motor function multi-dimensional intelligent evaluation method according to claim 3, characterized in that, In step 2-1, IMU data processing first applies a median filter to remove outliers and spike noise, then uses a Butterworth low-pass filter to remove high-frequency noise and muscle tremor interference, for angular velocity and acceleration signals, a Kalman filter is applied for signal smoothing and fusion to improve the accuracy of motion trajectory estimation; sEMG signal processing first applies band-pass filtering to remove motion artifacts and power frequency interference, then uses a notch filter to remove power supply noise, then applies an adaptive Wiener filter to remove possible crosstalk noise, and finally performs envelope extraction to obtain the time-varying curve of muscle activity intensity; EEG signal processing first applies band-pass filtering to retain the main EEG frequency band, then uses a notch filter to remove 50 / 60 Hz power supply noise, then uses independent component analysis ICA algorithm to identify and remove electrooculogram EOG artifacts and electromyography EMG artifacts, and finally applies spatial filtering to further improve signal quality; video data processing applies Gaussian filtering for image smoothing to reduce noise influence, uses background segmentation algorithm to extract foreground human contour, and applies time median filtering to reduce interframe jitter; In step 2-2, the start / end markers of the tasks in the data acquisition system are used to accurately extract the data segments corresponding to each task. For data without clear markers, an automatic segmentation algorithm based on energy threshold and motion pattern recognition is used to identify the start and end times of the tasks. Data with different sampling rates are resampled to a unified standard using an interpolation algorithm to ensure that the data of all modalities are strictly aligned on the time axis. Task data of different lengths are standardized into fixed-length sequences using dynamic time warping (DTW) or linear interpolation methods to facilitate batch training. In steps 2-3, IMU feature extraction includes time-domain features, frequency-domain features, statistical features, and kinematic features; sEMG feature extraction includes temporal features, frequency domain features, time-frequency features, and muscle synergy features; EEG feature extraction includes band energy, time-frequency features, complexity features, and functional connectivity features; visual feature extraction includes skeletal keypoint coordinates, joint angles, trajectory features, pose descriptors, and temporal motion features; all extracted features are normalized using Z-score or Min-Max normalization to eliminate dimensional differences between different sensors and individuals, and accelerate model convergence. In steps 2-4, a quality assessment network is designed to evaluate the quality of data for each modality: wherein, X m denotes the input of the quality assessment network QualityNet, representing the original data or pre-processed feature data from the m-th modality; ∈ [0, 1], denotes the quality score of the modality, which will be used for subsequent adaptive weight distribution; the quality assessment network is trained by a supervised learning method, with expert-labeled signal quality ratings as labels, to learn to identify noise levels, signal stability, and sensor shedding problems; in specific implementation, QualityNet adopts a lightweight CNN structure, including 3 convolutional layers and 2 fully connected layers, outputting a single quality score; wherein: the IMU quality assessment focuses on signal smoothness, motion coherence, and sensor drift degree; the sEMG quality assessment focuses on signal-to-noise ratio, baseline stability, and electrode contact quality; the EEG quality assessment focuses on artifact degree, impedance stability, and signal integrity; the visual quality assessment focuses on image clarity, occlusion degree, and key point detection confidence.

5. The method of claim 1, wherein, In step 3, the MMHF-Net core model processes heterogeneous data through a multi-level fusion strategy to achieve high-precision rehabilitation assessment. The MMHF-Net model includes a multimodal specific encoder, a cross-modal dynamic attention mechanism (CMDA), a hierarchical multi-level fusion module, an adaptive personalized assessment module, and a multi-task learning and joint optimization module. The workflow of step 3 is as follows: receiving the preprocessed multimodal feature sequence, and learning a deep feature vector that can comprehensively represent the patient's motor function status through the feature extraction, fusion, and encoding network inside the model. This model can not only handle the heterogeneity of different modal data, but also effectively capture the complex spatiotemporal dependencies between them. By introducing a robust mechanism, stable performance in real clinical environments is ensured. Finally, the module outputs a deep feature vector that integrates multimodal information, providing a foundation for the generation of subsequent multidimensional evaluation results.

6. The multi-modal hierarchical fusion network-based motor function multi-dimensional intelligent evaluation method according to claim 5, characterized in that, The multimodal specific encoders include inertial sensor encoders (IMU Encoders), surface electromyography (sEMG) encoders, electroencephalography (EEG) encoders, and visual encoders. The inertial sensor encoder (IMU Encoder) employs an improved temporal convolutional network (TCN), combining a dual-scale structure of LocalTCN (small receptive field for capturing fine-grained features) and GlobalTCN (large receptive field for capturing large-scale patterns). It also incorporates a motion pattern attention mechanism to automatically emphasize clinically significant motion patterns and introduces phase-aware processing and spatiotemporal resolution adaptive adjustment to improve the accuracy of motion trajectory feature extraction. The surface electromyography encoder sEMG Encoder adopts a hybrid architecture of CNN-GRU, and a frequency domain analysis branch and a muscle coordination analysis module to realize multi-domain fusion of time domain, frequency domain and nonlinear dynamics domain features, and to integrate dynamic fatigue detection and multi-muscle interaction modeling to comprehensively depict electromyographic activity characteristics. The electroencephalogram encoder EEG Encoder is based on an improved Transformer architecture, incorporates a spatial perception attention mechanism to capture spatial correlations between brain regions, synchronously introduces a functional connection network, and designs special processing channels for different electroencephalogram frequency bands to integrate a motor imagery classifier to evaluate differences in neural level motor control. The visual encoder Visual Encoder adopts a 3D-ResNet structure, combines a motion focusing module based on optical flow estimation, focuses on areas with significant motion changes, integrates anatomical prior attention, multi-scale temporal modeling, and posture estimation and motion analysis fusion to improve complex motion pattern recognition capability.

7. The multi-modal hierarchical fusion network-based motor function multi-dimensional intelligent evaluation method according to claim 5, characterized in that, The cross-modal dynamic attention mechanism CMDA dynamically adjusts weights based on quality scores of each modality, constructs bidirectional information interaction between modalities through multi-head cross-modal attention, designs an attention weight inter-layer transfer mechanism to guide high-level feature extraction by low-level attention patterns, and introduces contrast learning to promote alignment of semantic representations between modalities to solve modality heterogeneity problems, ensuring stable performance of the model in noisy or signal loss scenarios. The hierarchical multi-level fusion module implements differentiated fusion strategies at different levels of the network: early feature fusion performs convolutional fusion on low-level features of each modality at the early stage of feature extraction to realize basic information integration; mid-term feature fusion enhances feature discriminability by constructing interaction between modalities through an attention mechanism; late feature fusion realizes weighted fusion based on adaptive weights calculated based on quality and task relevance at the decision-making layer; the introduction of neural architecture search NAS automatically selects the optimal fusion path, enabling the model to dynamically optimize the fusion strategy according to task requirements and data characteristics; the multi-granularity fusion design captures full-spectrum features from low-level signal characteristics to high-level semantic information, and through progressive information distillation and multi-modal mutual learning mechanism, the model's expression ability and generalization performance are improved.

8. The multi-modal hierarchical fusion network-based motor function multi-dimensional intelligent evaluation method according to claim 5, characterized in that, The adaptive personalized evaluation module realizes precise adaptation of the model to individual differences based on a meta-learning framework, optimizes meta-parameters using an improved Reptile algorithm, and introduces personalized adaptation parameters at multiple levels of the network; meanwhile, it integrates static features and dynamic states of patients, realizes personalized adjustment of network features through feature linear modulation FiLM mechanism, introduces an external memory mechanism to store representations and adaptation strategies of similar patient cases, accelerates the personalized process of new patients, designs a progressive parameter update strategy to avoid catastrophic forgetting, and even in the case of limited patient data, the model can be quickly adapted through parameter fine-tuning to ensure evaluation accuracy.

9. The multi-modal hierarchical fusion network-based motor function multi-dimensional intelligent evaluation method according to claim 5, characterized in that, The multi-task learning and joint optimization module realizes joint prediction of "macroscopic-mesoscopic-microscopic" three-level evaluation indicators through a hierarchical prediction architecture: The macro-level output FMA, WMFT, MRS and Barthel index total score, the meso-level prediction of each scale sub-item score, the micro-level quantification of joint activity, muscle coordination, movement smoothness and muscle synergistic contraction mode fine-grained index, and the high-level prediction guiding the low-level prediction to enhance the consistency of evaluation; in order to improve the medical rationality and accuracy of prediction, a constraint enhanced loss function is designed, including the prediction loss of calculating the prediction error of each index, the scale correlation loss of using the correlation constraint of scale, the medical constraint loss of integrating the medical prior knowledge, and the evaluation consistency loss of ensuring the consistency of different modal predictions, and the feature sparsity, modal complementarity and time sequence consistency regularization strategies are introduced, the weight of each loss component is optimized through the validation set, and the overall loss function is formed to guide the model training.

10. A multi-dimensional intelligent evaluation system for motor function based on a multi-modal hierarchical fusion network, wherein the system is based on the multi-dimensional intelligent evaluation method for motor function based on a multi-modal hierarchical fusion network according to any one of claims 1-9, characterized in that, The multi-modal data input layer is used for receiving raw data from wearable inertial sensors IMU data, surface electromyography sEMG data, electroencephalogram EEG data and visual sensors; the data preprocessing and feature extraction module is used for completing the signal filtering, segmentation, alignment and feature extraction steps; the MMHF-Net core model integrates a multi-modal encoder, a cross-modal dynamic attention mechanism CMDA and a hierarchical fusion module, and is used for realizing efficient fusion of multi-modal data; and the multi-scale evaluation result output layer is used for generating macro, meso and micro three-level evaluation results.

Images