Cerebral stroke rehabilitation training action quality evaluation method and system based on multi-modal fusion

Abstract

The invention relates to a stroke rehabilitation training action quality evaluation method and system based on multi-modal fusion, and belongs to the technical field of intelligent rehabilitation and artificial intelligence. The system comprises a multi-modal data acquisition module, a video semantic and feature extraction module, a semantic-guided multi-modal fusion module, a special quantization module, a rehabilitation stage evaluation module and a multi-dimensional dynamic weighting evaluator. The method comprises the following steps: synchronously acquiring a training video, a surface electromyogram signal and inertial measurement data; extracting bottom visual features and high-level semantic features of the video; multi-modal feature fusion of semantic guidance is realized through a gating cross-modal attention mechanism; generating a special quantized value of the abnormal motion mode by using a conditional diffusion model under the condition of the fusion features; a rehabilitation stage is automatically evaluated based on historical data; and combining stage labels and real-time state dynamic weighting, and outputting a multi-dimensional comprehensive score. According to the invention, comprehensive, self-adaptive and intelligent evaluation of rehabilitation actions is realized, and evaluation objectivity and rehabilitation individuation level are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to a stroke rehabilitation training action quality evaluation method and system based on multi-modal fusion, belonging to the field of rehabilitation medicine and artificial intelligence, and is especially suitable for stroke rehabilitation training action quality evaluation based on multi-modal fusion. BACKGROUND

[0002] The rehabilitation of the limb motor function of a stroke patient is a long-term and highly personalized process. The quality of the rehabilitation training action is a key factor affecting the rehabilitation effect, and objective, accurate and comprehensive evaluation thereof is crucial for formulating a personalized treatment plan, dynamically adjusting the training difficulty and improving the rehabilitation efficiency.

[0003] At present, clinical rehabilitation evaluation mainly relies on the subjective observation and experience judgment of a rehabilitation therapist, and there are problems such as non-uniform evaluation standards, low efficiency and difficulty in realizing high-frequency quantitative tracking. Some existing automatic evaluation technologies are mostly based on single modal sensor data, such as using only an inertial measurement unit to analyze kinematics or using only a camera to perform posture estimation. These methods have obvious deficiencies: first, they lack the fusion of physiological signals such as surface electromyography, and cannot evaluate the neuromuscular control mode; second, they cannot understand the semantic information of the training task itself (for example, "what object was picked up" and "whether the operation sequence is correct"), resulting in evaluation remaining at the level of "whether the action is standard" and being unable to evaluate the functional goal of "whether the task is completed properly"; finally, the existing methods are mostly static evaluation frameworks, and cannot adaptively adjust the evaluation focus and standards according to the dynamically changing rehabilitation stage and real-time physiology of the patient, and the degree of individualization and intelligence is limited.

[0004] Therefore, there is an urgent need in the art for an intelligent evaluation scheme that can comprehensively fuse multi-dimensional information, deeply understand the task semantics, and dynamically adjust the evaluation strategy according to the patient state, in order to solve the problems of single evaluation dimension, functional deficiency and insufficient adaptability of the existing technology. SUMMARY

[0005] The present application aims to overcome the deficiencies of the prior art and provide a stroke rehabilitation training action quality evaluation method and system based on multi-modal fusion, in order to realize comprehensive, intelligent and individualized evaluation of the quality of rehabilitation training actions.

[0006] To achieve the above-mentioned purpose, the present application provides the following technical scheme:

[0007] The stroke rehabilitation training action quality evaluation system based on multi-modal fusion is composed of a multi-modal data acquisition module, a video semantic and feature extraction module, a semantic-guided multi-modal fusion module, a special item quantization module, a rehabilitation stage evaluation module and a multi-dimensional dynamic weighting evaluator. Figure 1 characterized in that it is composed of a multi-modal data acquisition module, a video semantic and feature extraction module, a semantic-guided multi-modal fusion module, a special item quantization module, a rehabilitation stage evaluation module and a multi-dimensional dynamic weighting evaluator.

[0008] The multimodal data acquisition module is used to simultaneously acquire patients' rehabilitation training videos, limb surface electromyography (sEMG) signals, and inertial measurement unit (IMU) data, and extract historical rehabilitation performance trend data.

[0009] The video semantics and feature extraction module is connected to the multimodal data acquisition module and is used to receive rehabilitation training videos, extract the low-level visual features of the videos through a shared visual backbone network, and parse the high-level semantic features of the videos through a hierarchical video semantic decoder. The high-level semantic features include scene context, target object information, and human-object interaction relationships.

[0010] The semantically guided multimodal fusion module connects the video semantic and feature extraction module and the multimodal data acquisition module, and is used to receive the low-level visual features, other modal coding features, and the high-level semantic features. This module adopts a hierarchical multimodal Transformer model, performs deep fusion of cross-modal features through a gated cross-modal attention mechanism, and uses the high-level semantic features to guide the fusion process, ultimately outputting a semantically enhanced fusion feature vector.

[0011] The specific quantization module is connected to the semantically guided multimodal fusion module and is used to drive the conditional diffusion model to generate specific quantization values ​​for abnormal motion patterns, based on the semantically enhanced fusion features.

[0012] The rehabilitation stage assessment module is connected to the multimodal data acquisition module. It is used to automatically assess and output the current rehabilitation stage label based on the patient's historical movement pattern quality, active joint range of motion, isolated movement frequency, and task completion rate through a classifier.

[0013] The multi-dimensional dynamic weighted evaluator connects the semantically guided multimodal fusion module, the specialized quantification module, and the rehabilitation stage assessment module. It receives the semantically enhanced fusion features, specialized quantification values, rehabilitation stage labels, and historical performance trend data. The evaluator sets a macro-weighting strategy based on the rehabilitation stage labels and adjusts the micro-weights according to real-time status. Finally, it calculates and outputs a multi-dimensional quality evaluation covering movement accuracy, movement pattern quality, body balance and stability, physiological load, task functionality, and a comprehensive total score.

[0014] The rehabilitation training video mentioned Surface electromyography (EMG) signals Inertial measurement data Historical rehabilitation performance data , Dimensions of rehabilitation characteristics; For time step, This refers to the number of video channels. , These are the height and width of the rehabilitation training video frames, respectively. For surface electromyography signal characteristics, This refers to the dimension of inertial measurement data.

[0015] Furthermore, the video semantics and feature extraction module is implemented using a shared Vision Transformer (ViT) backbone network with low-level feature branches and high-level semantic branches, which integrates rehabilitation training videos. The video frame sequence is used as input; the low-level feature branch consists of the intermediate layer of ViT. This is obtained by concatenating a linear layer, which processes the video frame sequence through the first... The ViT layer transforms to obtain a sequence of visual tokens. Then linearize to low-level visual features ,in Number of blocks The lower-level visual feature dimension; the higher-level semantic branch is composed of ViT connected to a three-level Transformer decoder, wherein the first-level Transformer decoder performs self-attention and cross-attention calculations on the final output visual token sequence of ViT, and outputs aggregated features. The second-stage Transformer decoder takes the learned object feature vectors as input and interacts with the final output visual token sequence of ViT, outputting the category embedding and temporal bounding box object tokens for each object. The third-level Transformer decoder uses human joint tokens and object tokens. Using the input as input, the spatiotemporal correlation is calculated through the attention module to generate human-object interaction features. ;in, The number of objects detected. For scene semantic feature dimension, For the semantic feature dimension of objects, The interactive semantic feature dimension; the human joint token is generated using an attention module from... The tokens corresponding to the joints extracted from the learned joint feature vectors.

[0016] Furthermore, the hierarchical multimodal Transformer model of the semantically guided multimodal fusion module is composed of an encoding layer, a gated cross-modal attention layer, a temporal modeling layer, and an aggregated output layer connected in series;

[0017] The encoding layer is a linear layer that encodes the underlying visual features. Surface electromyography characteristics Inertial measurement characteristics Each dimension is projected through an independent linear layer. The corresponding encoded feature vector is obtained. , , The and They are respectively , Output encoded using 1D-CNN;

[0018] For any modality features , The working principle of the gated cross-modal attention layer is as follows:

[0019] (STEP1) Generate the corresponding query matrix using the weight matrix. Key matrix Value matrix ;

[0020] (STEP2) High-level semantic features and Perform concatenation and linear transformation to generate semantic condition vectors. ;

[0021] (STEP3) Calculate the standard attention weights: ; for The dimension;

[0022] (STEP4) Calculate the gating vector: ;in, For splicing, It is a multilayer sensing network;

[0023] (STEP5) Calculate the standard attention weights after gating: ;in For element-wise multiplication, For the Sigmoid function;

[0024] (STEP6) Output the token sequence after modal update. ;

[0025] The temporal modeling layer is a standard Transformer encoder, which sums the token sequences after all modal updates and uses them as input.

[0026] The aggregated output layer is constructed as an average pooling layer, yielding a fixed-dimensional semantically enhanced fusion feature vector. .

[0027] Furthermore, the specific quantification module is a Conditional Denoising Diffusion Probability Model (CDDPM); its forward diffusion process is a Markov chain with fixed variance, progressively moving towards real expert scores. Add noise to get Noise fraction at time Its reverse denoising network It is a U-Net structured neural network whose input is Diffusion Step Embedding and conditional features Its loss function is ,in Use realistic noise; during training, use standard Gaussian noise. Begin, execute The reverse denoising iteration is repeated to finally obtain the generated specific quantization value. .

[0028] Furthermore, the rehabilitation stage assessment module is a lightweight classifier that takes historical rehabilitation performance data as input. Output rehabilitation stage tags ;in, For Brunnstrom installment plans.

[0029] Furthermore, the multi-dimensional dynamic weighted evaluator includes:

[0030] Weighting strategy unit, used to determine the weighting strategy based on the rehabilitation stage label. Read the macro weights of each evaluation dimension from the pre-set strategy table;

[0031] The micro-dynamic adjustment scoring generation unit is a multilayer perceptron, used to generate the semantically enhanced fusion feature vector. Specialized quantitative values The system takes macro-level weights as input and outputs a six-dimensional vector encompassing motion accuracy, movement pattern quality, body balance and stability, physiological load, task functionality, and overall score. .

[0032] As a multilayer sensing network, based on rehabilitation stage labels The macro-weights are read from the pre-defined policy table and then fused with the semantically enhanced feature vector. Specialized quantitative values As input to the multilayer sensing network, the output is a six-dimensional vector encompassing motion accuracy, movement pattern quality, body balance and stability, physiological load, task functionality, and overall score. .

[0033] Furthermore, the preset strategy table is a normalized table after experts score the importance of six indicators corresponding to the Brunnstrom stage: movement accuracy, movement pattern quality, body balance and stability, physiological load, task functionality, and overall score.

[0034] Accordingly, the present invention also provides a method for assessing the quality of stroke rehabilitation training movements based on multimodal fusion, characterized by comprising the following steps:

[0035] S1: Simultaneously collect multimodal data, including rehabilitation training videos, surface electromyography signals, inertial measurement data, and acquire historical rehabilitation performance data;

[0036] S2: Process rehabilitation training videos, extract low-level visual features describing the appearance of movement, and parse out high-level semantic features that include scene context, target objects, and human-object interaction relationships.

[0037] S3: Perform semantically guided multimodal fusion of low-level visual features, encoding features of other modal data, and high-level semantic features; adopt a hierarchical multimodal Transformer model, achieve deep cross-modal fusion through a gated cross-modal attention mechanism, and use high-level semantic features to modulate attention calculation to generate semantically enhanced fusion feature vectors;

[0038] S4: Using the semantically enhanced fusion features as conditions, generate specific quantization values ​​for quantifying abnormal motion patterns using a conditional diffusion model;

[0039] S5: Based on historical rehabilitation performance data, the classifier automatically assesses the current rehabilitation stage corresponding to the patient's multimodal data to obtain a rehabilitation stage label;

[0040] S6: Integrate the semantically enhanced fusion features, specific quantitative values, and rehabilitation stage labels, implement a macro-weighting strategy based on the rehabilitation stage, and perform micro-dynamic adjustments based on real-time status to finally calculate a multi-dimensional comprehensive score for movement quality.

[0041] S7: Outputs rehabilitation stage labels and multi-dimensional comprehensive scores of movement quality to assist doctors in assessing the quality of rehabilitation training movements for stroke patients.

[0042] Furthermore, the multimodal data mentioned in step S1 is aligned to a unified time base using either hardware synchronization signals or software timestamps. The video data was downsampled and normalized in size, and the IMU and sEMG data were bandpass filtered, normalized, and segmented into segments aligned with the video frame rate.

[0043] An electronic device includes at least one processor; and a memory communicatively connected to said at least one processor; wherein,

[0044] The memory stores a computer program that is executed by the at least one processor, which enables the at least one processor to perform the above-described method for assessing the quality of stroke rehabilitation training movements based on multimodal fusion.

[0045] Finally, the present invention also discloses a computer-readable storage medium storing computer instructions for causing a processor to execute the above-described method for assessing the quality of stroke rehabilitation training movements based on multimodal fusion.

[0046] Compared with existing technologies, this invention has the following significant advantages: 1) By fusing multimodal data from vision, inertia, and electromyography, it achieves comprehensive perception of kinematics and neuromuscular control; 2) It designs a hierarchical multimodal fusion architecture and a gated cross-modal attention mechanism, which can adaptively adjust the information flow between different modalities, improving the robustness and effectiveness of feature fusion; 3) It innovatively uses a conditional diffusion model to perform stable and noise-resistant specific quantification of abnormal movement patterns that are difficult to quantitatively assess clinically; 4) It constructs a complete assessment system that includes automatic assessment of the rehabilitation stage and multidimensional dynamic weighted assessment, realizing end-to-end intelligent processing from data collection to comprehensive scoring; 5) Through a macro-weighting strategy based on the rehabilitation stage and a micro-dynamic adjustment based on real-time status, the assessment criteria can adapt to the patient's rehabilitation process and immediate status, achieving truly personalized and intelligent assessment, which can effectively assist in rehabilitation decision-making and training program adjustment. Attached Figure Description

[0047] To make the objectives and technical solutions of this invention clearer, the following figures are provided for illustration:

[0048] Figure 1 This is an architecture diagram of the stroke rehabilitation training movement quality assessment system based on multimodal fusion in this invention;

[0049] Figure 2 This is a schematic diagram illustrating the working principle of the video semantics and feature extraction module in Embodiment 1 of the present invention;

[0050] Figure 3 This is a schematic diagram illustrating the working principle of the semantically guided multimodal fusion module in Embodiment 1 of the present invention.

[0051] Figure 4 This is a flowchart of the stroke rehabilitation training movement quality assessment method based on multimodal fusion in this invention;

[0052] Figure 5 This is a schematic diagram of the electronic device in Embodiment 3 of the present invention. Detailed Implementation

[0053] To make the objectives and technical solutions of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0054] Example 1: Rehabilitation training is a key means to restore patients' motor function, but traditional rehabilitation assessment mainly relies on the therapist's subjective observation, which has the following prominent problems: different therapists have large differences in their scores for the same movement; key indicators such as abnormal synergistic movements and isolated movements are difficult to measure accurately; a complete Brunnstrom stage assessment takes a long time; and it is impossible to achieve real-time monitoring and dynamic adjustment of daily training.

[0055] Taking the rehabilitation medicine department of a top-tier hospital as an example, with 120 beds and approximately 800 stroke patients admitted annually, there are only 15 rehabilitation therapists. Therapists need to assess 40-50 patients daily, resulting in an extremely heavy workload and difficulty in ensuring the consistency and timeliness of assessments. Especially during patients' daily repetitive training, therapists cannot supervise the entire process, leading to inconsistent training quality and affecting rehabilitation outcomes. To address these problems, this invention provides "a multimodal fusion-based stroke rehabilitation training movement quality assessment system," aiming to achieve automated, objective, and continuous assessment of the quality of rehabilitation training movements.

[0056] In this embodiment, the system is equipped with the following hardware: multiple high-definition cameras (for acquiring training videos), a wireless surface electromyography (sEMG) sensor (for acquiring sEMG signals), and an inertial measurement unit (integrated in a wristband or strap for acquiring IMU data). All devices are aligned via hardware synchronization modules or software timestamps to ensure the temporal consistency of multimodal data. The acquired data is transmitted in real time to a backend server for processing via a network.

[0057] Before the system is started, a personal file needs to be created for each patient in the training set and historical data tags need to be initialized, including: (a) patient basic information: age, gender, stroke type, affected side, medical history, etc.; (b) historical rehabilitation performance data: score records of each dimension in past training, joint range of motion trend, isolated movement frequency, task completion rate, etc.; (c) rehabilitation stage tags: initially manually marked by the therapist according to the Brunnstrom stage, and subsequently automatically updated by the system.

[0058] Specifically, in combination Figure 1 The system consists of a multimodal data acquisition module (1), a video semantic and feature extraction module (2), a semantically guided multimodal fusion module (3), a specialized quantification module (4), a rehabilitation stage assessment module (5), and a multidimensional dynamic weighted evaluator (6);

[0059] The multimodal data acquisition module (1) is used to simultaneously acquire patients' rehabilitation training videos, limb surface electromyography (sEMG) signals, inertial measurement data (IMU), and extract historical rehabilitation performance trend data.

[0060] The video semantic and feature extraction module (2) is connected to the multimodal data acquisition module (1) and is used to receive rehabilitation training videos, extract the low-level visual features of the videos through a shared visual backbone network, and parse the high-level semantic features of the videos through a hierarchical video semantic decoder. The high-level semantic features include scene context, target object information and human-object interaction relationship.

[0061] The semantically guided multimodal fusion module (3) connects the video semantic and feature extraction module (2) and the multimodal data acquisition module (1), and is used to receive the low-level visual features, other modal coding features and the high-level semantic features. This module adopts a hierarchical multimodal Transformer model, performs deep fusion of cross-modal features through a gated cross-modal attention mechanism, and uses the high-level semantic features to guide the fusion process, and finally outputs a semantically enhanced fusion feature vector.

[0062] The specific quantization module (4) is connected to the semantically guided multimodal fusion module (3) and is used to drive the conditional diffusion model to generate specific quantization values ​​for abnormal motion patterns based on the semantically enhanced fusion features.

[0063] The rehabilitation stage assessment module (5) is connected to the multimodal data acquisition module (1) and is used to automatically assess and output the current rehabilitation stage label based on the patient's historical movement pattern quality, active joint range of motion, isolated movement frequency and task completion degree through a classifier.

[0064] The multi-dimensional dynamic weighted evaluator (6) is connected to the semantically guided multimodal fusion module (3), the specialized quantification module (4), and the rehabilitation stage assessment module (5). It is used to receive the semantically enhanced fusion features, specialized quantification values, rehabilitation stage labels, and historical performance trend data. The evaluator sets a macro-weighting strategy based on the rehabilitation stage labels and adjusts the micro-weights according to the real-time status. Finally, it calculates and outputs a multi-dimensional quality evaluation covering movement accuracy, movement pattern quality, body balance and stability, physiological load, task functionality, and overall score.

[0065] The rehabilitation training video mentioned Surface electromyography (EMG) signals Inertial measurement data Historical rehabilitation performance data , For the past The dimensions of rehabilitation features extracted from each training session, including the mean, variance, and range of motion of the movement patterns. For time step, This refers to the number of video channels. , These are the height and width of the rehabilitation training video frames, respectively. For surface electromyography signal characteristics, This refers to the dimension of inertial measurement data.

[0066] Furthermore, combined with Figure 2 The video semantics and feature extraction module (2) is implemented using a shared Vision Transformer (ViT) backbone network with low-level feature branches and high-level semantic branches to extract rehabilitation training videos. The video frame sequence is used as input; the low-level feature branch consists of the intermediate layer of ViT. This is obtained by concatenating a linear layer, which processes the video frame sequence through the first... The ViT layer transforms to obtain a sequence of visual tokens. Then linearize to low-level visual features ,in Number of blocks The lower-level visual feature dimension; the higher-level semantic branch is composed of ViT connected to a three-level Transformer decoder, wherein the first-level Transformer decoder performs self-attention and cross-attention calculations on the final output visual token sequence of ViT, and outputs aggregated features. The second-stage Transformer decoder takes the learned object feature vectors as input and interacts with the final output visual token sequence of ViT, outputting the category embedding and temporal bounding box object tokens for each object. The third-level Transformer decoder uses human joint tokens and object tokens. Using the input as input, the spatiotemporal correlation is calculated through the attention module to generate human-object interaction features. ;in, The number of objects detected. For scene semantic feature dimension, For the semantic feature dimension of objects, The interactive semantic feature dimension; the human joint token is generated using an attention module from... The tokens corresponding to the joints extracted from the learned joint feature vectors.

[0067] Among them, the learned joint feature vectors are spatiotemporal skeleton maps of human joints extracted from rehabilitation training videos using deep learning networks; the learned object feature vectors are feature vectors extracted from cropped object images in rehabilitation training videos using deep learning networks.

[0068] Furthermore, combined with Figure 3 The semantically guided multimodal fusion module (3) consists of a hierarchical multimodal Transformer model composed of an encoding layer (31), a gated cross-modal attention layer (32), a temporal modeling layer (33), and an aggregated output layer (34) connected in series.

[0069] The encoding layer (31) is a linear layer that encodes the underlying visual features. Surface electromyography characteristics Inertial measurement characteristics Each dimension is projected through an independent linear layer. The corresponding encoded feature vector is obtained. , , The and They are respectively , Output encoded using 1D-CNN;

[0070] For any modality features , The working principle of the gated cross-modal attention layer (32) is as follows:

[0071] (STEP1) Generate the corresponding query matrix using the weight matrix. Key matrix Value matrix ;

[0072] (STEP2) High-level semantic features and Perform concatenation and linear transformation to generate semantic condition vectors. ;

[0073] (STEP3) Calculate the standard attention weights: ; for The dimension;

[0074] (STEP4) Calculate the gating vector: ;in, For splicing, It is a multilayer sensing network;

[0075] (STEP5) Calculate the standard attention weights after gating: ;in For element-wise multiplication, For the Sigmoid function;

[0076] (STEP6) Output the token sequence after modal update. ;

[0077] The aforementioned temporal modeling layer (33) is a standard Transformer encoder, which sums the token sequences after all modal updates and uses them as input;

[0078] The aggregated output layer (34) is an average pooling layer, which yields a fixed-dimensional semantically enhanced fusion feature vector. .

[0079] Furthermore, the specific quantification module (4) is a conditional denoising diffusion probability model (CDDPM); its forward diffusion process is a Markov chain with fixed variance, gradually moving towards real expert scores. Add noise to get Noise fraction at time Its reverse denoising network It is a U-Net structured neural network whose input is Diffusion Step Embedding and conditional features Its loss function is ,in This is real noise. During training, real expert scores are incorporated through forward diffusion. Add noise to generate standard Gaussian noise Then from standard Gaussian noise Initially, an inverse denoising network was used. implement The reverse denoising iteration is repeated to finally obtain the generated specific quantization value. During the test, standard Gaussian noise was randomly generated. Then, an inverse denoising network is used. implement The reverse denoising iteration is repeated to finally obtain the generated specific quantization value. .

[0080] Furthermore, the rehabilitation stage assessment module (5) is a lightweight classifier that takes historical rehabilitation performance data as input. Output rehabilitation stage tags ;in, For Brunnstrom installment plans.

[0081] Furthermore, the multi-dimensional dynamic weighted evaluator (6) includes: a weighting strategy unit (61) for weighting the rehabilitation stage labels. The macro-weights of each evaluation dimension are read from the pre-set strategy table; the micro-dynamic adjustment scoring generation unit (62) is a multi-layer perceptual network used to generate the semantically enhanced fusion feature vector. Specialized quantitative values The system takes macro-level weights as input and outputs a six-dimensional vector encompassing motion accuracy, movement pattern quality, body balance and stability, physiological load, task functionality, and overall score. .

[0082] Furthermore, the preset strategy table is a normalized table after experts score the importance of six indicators corresponding to the Brunnstrom stage: movement accuracy, movement pattern quality, body balance and stability, physiological load, task functionality, and overall score.

[0083] This system adopts an end-to-end training strategy, with an overall loss function. It consists of the following two parts, designed to simultaneously optimize feature learning and multi-task evaluation:

[0084] (a) Uncertainty-weighted contrastive learning loss The semantically enhanced fusion feature vector output by the semantically guided multimodal fusion module (3) This loss dynamically adjusts the sample weights based on the uncertainty of the model's predictions, enhancing the model's robustness to noise and individual differences. Its expression is:

[0085] ;

[0086] in, For InfoNCE comparison loss, The feature embeddings (i.e. semantically enhanced fused feature vectors) for patient sample pairs in the training set. ), The uncertainty score for the model's prediction of sample pairs.

[0087] (b) Multi-task learning loss based on homoscedasticity uncertainty The output of the six evaluation dimensions (action accuracy, movement pattern quality, body balance and stability, physiological load, task functionality, and overall score) applied to the multi-dimensional dynamic weighted evaluator (6) is used. This loss automatically learns the uncertainty of each task and balances its contribution to the total loss accordingly, solving the weight allocation problem in multi-task learning. Its expression is:

[0088] ;

[0089] in, For the first Regression loss in multiple dimensions (such as smoothed L1 loss). For the model is the first The homoscedasticity uncertainty parameters learned from each task.

[0090] The system's total loss function is a weighted sum of the two: ;in To balance the hyperparameters.

[0091] In practical deployment, the system in this embodiment can automatically complete the entire process from multimodal data acquisition, semantic understanding, feature fusion, specialized quantification, stage assessment to final multidimensional scoring. Therapists can view the rehabilitation stage suggestions, detailed dimensional scoring radar charts, and historical trend analysis automatically generated by the system through the terminal, thereby quickly and accurately grasping the patient's rehabilitation progress, formulating or adjusting personalized training plans, and greatly improving assessment efficiency and the scientific nature of rehabilitation management.

[0092] Example 2: For the scenario described in Example 1, this invention also provides "a method for assessing the quality of stroke rehabilitation training movements based on multimodal fusion." This method can be used in conjunction with the system described in Example 1, or it can be deployed independently on a computing platform with corresponding processing capabilities, achieving full automation from data acquisition to intelligent assessment. The method has a clear flow and well-defined steps, facilitating clinical deployment and program implementation.

[0093] Combination Figure 4 The method specifically includes the following steps:

[0094] S1: Synchronous acquisition and preprocessing of multimodal data.

[0095] Simultaneous collection of patients' rehabilitation training videos Surface electromyography (EMG) signals Inertial measurement data Historical rehabilitation performance data All sensor data is aligned to a unified time base via hardware synchronization signals or high-precision software timestamps. The video data was downsampled and normalized in size; the IMU and sEMG data were bandpass filtered, amplitude normalized, and segmented into temporal segments aligned with the video frame rate. Historical rehabilitation performance data included rehabilitation features such as the mean and variance of movement pattern quality, active joint range of motion, isolated movement frequency, and task completion rate extracted from several past training sessions.

[0096] S2: Video semantics and feature extraction.

[0097] rehabilitation training videos Input is a shared Vision Transformer (ViT) backbone network. This is achieved through the intermediate layer (the...) of the ViT... Extracting visual token sequences from layers The underlying visual features are obtained through linear projection. This is used to describe the appearance and posture information of motion. Simultaneously, the deep output of ViT is processed by a hierarchical video semantic decoder consisting of a three-level Transformer decoder, sequentially parsing out high-level semantic features: scene context features. Represents the training environment and object layout; target object token. This includes the category embedding and temporal bounding box for each detected object; human-object interaction features. It depicts the spatiotemporal relationship between human joints and objects.

[0098] S3: Semantic-guided multimodal feature fusion.

[0099] underlying visual features Surface electromyography characteristics (Depend on (Obtained via 1D-CNN encoding) Inertial measurement features (Depend on (The encoded features) are projected onto a unified dimension through independent linear layers to obtain the corresponding encoded feature vectors. , , These encoded features, along with high-level semantic features, are input into a hierarchical multimodal Transformer model. The model first uses semantic conditional vectors to modulate the attention weights between different modalities in a gated cross-modal attention layer, achieving adaptive and robust cross-modal information fusion. Then, in the temporal modeling layer, a standard self-attention mechanism is used to capture long-term dependencies in action sequences. Finally, a fixed-dimensional semantically enhanced fusion feature vector is obtained through an aggregated output layer (average pooling). .

[0100] S4: Quantitative analysis of abnormal motion patterns based on conditional diffusion model.

[0101] semantically enhanced fusion feature vector As a condition, the input conditional denoising diffusion probability model (CDDPM) is used. During the training phase, this model learns the probability distribution of rehabilitation experts' ratings of abnormal movement patterns; during the inference phase, through a multi-step iterative denoising process, a stable and noise-resistant specific quantification value is generated from the learned distribution. It is used to objectively measure the severity of abnormal coordinated movements and other patterns in a patient's actions.

[0102] S5: Automatic assessment during the rehabilitation phase.

[0103] Historical rehabilitation performance data Input a lightweight classifier (such as a support vector machine or gradient boosting decision tree), which automatically predicts the patient's current rehabilitation stage based on recent performance in areas such as quality of movement patterns, range of motion of active joints, frequency of isolated movements, and task completion, and outputs a discrete rehabilitation stage label. (e.g., corresponding to Brunnstrom stages III, IV, V, etc.).

[0104] S6: Multi-dimensional dynamic weighted evaluation.

[0105] semantically enhanced fusion feature vectors Specialized quantitative values Rehabilitation stage labels Input a multi-dimensional dynamic weighted evaluator. This evaluator first depends on... The system reads the macro-level weights for each evaluation dimension from a pre-defined strategy table; then, it performs micro-level dynamic adjustments based on real-time status indicators. Finally, based on the dynamically adjusted weights, the evaluator calculates and outputs a quality evaluation vector covering six dimensions: movement accuracy, movement pattern quality, body balance and stability, physiological load, task functionality, and overall score. .

[0106] S7: Evaluation results output and application.

[0107] Output rehabilitation stage labels and multi-dimensional quality evaluation vectors Furthermore, it can generate visual reports (such as radar charts and trend curves). The assessment results are pushed to the doctor's or therapist's terminal in real time, assisting them in accurately assessing the quality of the patient's rehabilitation training movements and providing data-driven decision support for the development and dynamic adjustment of personalized rehabilitation plans.

[0108] This method, through the above seven steps, achieves a comprehensive, intelligent, and personalized assessment of the quality of stroke rehabilitation training movements, effectively overcoming the limitations of traditional subjective assessment and single-modal automated assessment.

[0109] Example 3: For the scenario in Example 1, Figure 5 A schematic diagram of an electronic device (90) that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers.

[0110] Electronic devices can also refer to various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or claimed herein.

[0111] like Figure 5 As shown, the electronic device (90) includes at least one processor (91) and a memory, such as a read-only memory (ROM) (92) or a random access memory (RAM) (93), which is communicatively connected to the at least one processor (91). The memory stores computer programs executable by the at least one processor. The processor (91) can perform various appropriate actions and processes based on the computer programs stored in the ROM (92) or loaded from storage units (98) into the RAM (93). The RAM (43) may also store various programs and data required for the operation of the electronic device (90). The processor (91), ROM (42), and RAM (43) are interconnected via a bus (94). An input / output (I / O) interface (95) is also connected to the bus (94).

[0112] Multiple components in the electronic device (90) are connected to the I / O interface (95), including: input units (96), such as keyboards, mice, etc.; output units (97), such as various types of displays, speakers, etc.; storage units (98), such as disks, optical disks, etc.; and communication units (99), such as network cards, modems, wireless transceivers, etc. The communication unit (99) allows the electronic device (90) to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0113] The processor (91) can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processors (91) include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The processor (91) performs the various methods and processes described above, such as a multimodal fusion-based method for assessing the quality of stroke rehabilitation training movements.

[0114] In some embodiments, the method for assessing the quality of stroke rehabilitation training movements based on multimodal fusion can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as a storage unit (98). In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device (90) via a ROM (92) and / or a communication unit (99). When the computer program is loaded into RAM (93) and executed by a processor (91), one or more steps of the method for assessing the quality of stroke rehabilitation training movements based on multimodal fusion described above can be performed. Alternatively, in other embodiments, the processor (91) can be configured to perform the method for assessing the quality of stroke rehabilitation training movements based on multimodal fusion by any other suitable means (e.g., by means of firmware).

[0115] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0116] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0117] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0118] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0119] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0120] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0121] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made to it in form and detail without departing from the scope defined by the claims of the present invention.

Claims

1. A stroke rehabilitation training movement quality assessment system based on multimodal fusion, characterized in that, It consists of a multimodal data acquisition module (1), a video semantic and feature extraction module (2), a semantically guided multimodal fusion module (3), a specialized quantification module (4), a rehabilitation stage assessment module (5), and a multidimensional dynamic weighted evaluator (6); The multimodal data acquisition module (1) is used to simultaneously acquire patients' rehabilitation training videos, limb surface electromyography (sEMG) signals and inertial measurement data (IMU), and extract historical rehabilitation performance trend data; The video semantics and feature extraction module (2) is connected to the multimodal data acquisition module (1) and is used to receive rehabilitation training videos and extract the underlying visual features of the videos through a shared visual backbone network. The high-level semantic features of the video are analyzed by a hierarchical video semantic decoder. These high-level semantic features include scene context, target object information, and human-object interaction relationships. The semantically guided multimodal fusion module (3) connects the video semantic and feature extraction module (2) and the multimodal data acquisition module (1) and is used to receive the low-level visual features, other modal coding features and the high-level semantic features; it adopts a hierarchical multimodal Transformer model, performs deep fusion of cross-modal features through a gated cross-modal attention mechanism, and uses the high-level semantic features to guide the fusion process, and finally outputs a semantically enhanced fusion feature vector; The specific quantization module (4) is connected to the semantically guided multimodal fusion module (3) and is used to drive the conditional diffusion model to generate specific quantization values ​​for abnormal motion patterns based on the semantically enhanced fusion features. The rehabilitation stage assessment module (5) is connected to the multimodal data acquisition module (1) and is used to automatically assess and output the current rehabilitation stage label based on the patient's historical movement pattern quality, active joint range of motion, separation movement frequency and task completion degree through a classifier. The multidimensional dynamic weighted evaluator (6) is connected to the semantically guided multimodal fusion module (3), the specialized quantification module (4), and the rehabilitation stage evaluation module (5). It is used to receive the semantically enhanced fusion feature vector, the specialized quantification value, the rehabilitation stage label, and historical performance trend data. Based on the rehabilitation stage label, it sets a macro-weight strategy and adjusts the micro-weight according to the real-time status. Finally, it calculates and outputs a multidimensional quality evaluation covering movement accuracy, movement pattern quality, body balance and stability, physiological load, task functionality, and overall score.

2. The stroke rehabilitation training movement quality assessment system based on multimodal fusion according to claim 1, characterized in that, The video semantics and feature extraction module (2) adopts a shared Vision Transformer backbone network with a low-level feature branch and a high-level semantic branch; The low-level feature branch consists of an intermediate layer of ViT connected in series with a linear layer, which is used to convert the video frame sequence into low-level visual features. The high-level semantic branch consists of ViT connected in series with a three-level Transformer decoder. The first-level Transformer decoder performs self-attention and cross-attention calculations on the final output visual token sequence of ViT and outputs aggregated features. The second-level Transformer decoder takes the learned object feature vectors as input and interacts with the final output visual token sequence of ViT to output the category embedding and temporal bounding box object tokens for each object. The third-level Transformer decoder takes human joint tokens and object tokens as input and calculates their spatiotemporal correlation through the attention module to generate human-object interaction features.

3. The stroke rehabilitation training movement quality assessment system based on multimodal fusion according to claim 1, characterized in that, The semantically guided multimodal fusion module (3) consists of a hierarchical multimodal Transformer model composed of an encoding layer (31), a gated cross-modal attention layer (32), a temporal modeling layer (33), and an aggregated output layer (34) connected in series. The coding layer (31) is a linear layer that projects the bottom-level visual features, surface electromyography features and inertial measurement features to a unified dimension. The gated cross-modal attention layer (32) generates a semantic condition vector using the high-level semantic features, and dynamically adjusts the attention weights between different modalities through the learnable gated vector to achieve cross-modal feature interaction and fusion, and outputs the token sequence after modal update; The temporal modeling layer (33) is a standard Transformer encoder that models the temporal dependencies of the token sequences after all modal updates are fused. The aggregated output layer (34) is an average pooling layer that aggregates the time-series token sequences into a fixed-dimensional semantically enhanced fusion feature vector.

4. The stroke rehabilitation training movement quality assessment system based on multimodal fusion according to claim 1, characterized in that, The specific quantization module (4) is a conditional denoising diffusion probability model. Its forward diffusion process is a Markov chain with a fixed variance, and its reverse denoising network is a U-Net structure neural network with a training loss function of mean square error loss. Noise is added through the forward diffusion process, and the reverse denoising network is denoised with the semantically enhanced fusion feature vector as a condition, and finally, a specific quantization value is generated.

5. The stroke rehabilitation training movement quality assessment system based on multimodal fusion according to claim 1, characterized in that, The multi-dimensional dynamic weighted evaluator (6) includes: The weighting strategy unit (61) is used to read the macro weights of each assessment dimension from the preset strategy table according to the rehabilitation stage label; The micro-dynamic adjustment scoring generation unit (62) is a multilayer perceptual network used to take the semantically enhanced fusion feature vector, specific quantization value and macro weight as input, and output action accuracy, movement pattern quality, body balance and stability, physiological load, task functionality and comprehensive total score.

6. The stroke rehabilitation training movement quality assessment system based on multimodal fusion according to claim 5, characterized in that, The pre-set strategy table is a normalized table based on the scores given by experts according to the importance of six indicators: movement accuracy, movement pattern quality, body balance and stability, physiological load, task functionality, and overall score corresponding to the Brunnstrom stage.

7. A method for assessing the quality of stroke rehabilitation training movements based on multimodal fusion, characterized in that, Includes the following steps: S1: Simultaneously collect multimodal data, including rehabilitation training videos, surface electromyography signals, inertial measurement data, and acquire historical rehabilitation performance data; S2: Process rehabilitation training videos, extract low-level visual features describing the appearance of movement, and parse out high-level semantic features including scene context, target objects, and human-object interaction relationships. S3: Perform semantically guided multimodal fusion of low-level visual features, encoding features of other modal data, and high-level semantic features; adopt a hierarchical multimodal Transformer model, achieve deep cross-modal fusion through a gated cross-modal attention mechanism, and use high-level semantic features to modulate attention calculation to generate semantically enhanced fusion feature vectors; S4: Using the semantically enhanced fusion features as conditions, generate specific quantization values ​​for quantifying abnormal motion patterns using a conditional diffusion model; S5: Based on historical rehabilitation performance data, the classifier automatically assesses the current rehabilitation stage corresponding to the patient's multimodal data to obtain a rehabilitation stage label; S6: Integrate the semantically enhanced fusion features, specific quantitative values, and rehabilitation stage labels, implement a macro-weighting strategy based on the rehabilitation stage, and perform micro-dynamic adjustments based on real-time status to finally calculate a multi-dimensional comprehensive score for movement quality. S7: Outputs rehabilitation stage labels and multi-dimensional comprehensive scores of movement quality to assist doctors in assessing the quality of rehabilitation training movements for stroke patients.

8. The method for assessing the quality of stroke rehabilitation training movements based on multimodal fusion according to claim 7, characterized in that, The multimodal data described in step S1 uses hardware synchronization signals or software timestamp alignment to bring all sensor data to a unified time reference, and performs downsampling and size normalization on video data, bandpass filtering and normalization on IMU and sEMG data, and segments them into segments aligned with the video frame rate.

9. The method for assessing the quality of stroke rehabilitation training movements based on multimodal fusion according to claim 7, characterized in that, The gated cross-modal attention mechanism described in step S3 is implemented through the following steps: (STEP1) Calculate the query matrix, key matrix, and value matrix based on the feature vectors of each modality; (STEP2) Concatenate the high-level semantic features and linearly transform them into a semantic condition vector; (STEP3) Calculate the standard attention weights; (STEP4) Based on the semantic condition vector and the aggregation information of query and key, a learnable gating vector is calculated through a multilayer perceptron. (STEP5) Multiply the standard attention weights element-wise with the gated vector modulated by the Sigmoid function to obtain the gated attention weights; (STEP6) is used to weight the aggregated value matrix to update the modal token sequence.

10. An electronic device and a computer-readable storage medium, characterized in that, The electronic device includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the stroke rehabilitation training movement quality assessment method based on multimodal fusion as described in any one of claims 7-8; the computer-readable storage medium stores computer instructions for causing the processor to implement the stroke rehabilitation training movement quality assessment method based on multimodal fusion as described in any one of claims 7-8 when executed.

Images