Stroke patient upper limb movement scoring system, method and equipment oriented to bedridden evaluation scene

Abstract

The invention discloses a stroke patient upper limb motion scoring system, method and device for a bedridden evaluation scene, and relates to the field of NIHSS scoring, and the system comprises the steps that a multi-modal data acquisition module collects video optical flow data and inertial motion data; the feature extraction module performs feature extraction to obtain a visual modal representation vector and an action modal representation vector; the feature fusion module adopts a CrossAttention attention mechanism to carry out cross calculation on attention so as to obtain fused visual features and fused action features; the scoring and uncertainty prediction module is combined with an Evidenal Deep Learning framework and a Dempster-Shafer evidence fusion theory, so that an upper limb movement scoring value and a corresponding confidence interval are determined. Visual and inertial motion data are effectively fused, the limitation of a single mode is overcome, and accurate evaluation of the upper limb motion function in a bedridden scene is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of NIHSS score, in particular to a stroke patient upper limb movement scoring system, method and device for bed assessment scene. BACKGROUND

[0002] The National Institutes of Health Stroke Scale (NIHSS) score is an important tool for clinically assessing the degree of neurological function impairment of stroke patients, and the upper limb movement function score as a key indicator directly affects the quantitative assessment of the degree of motor impairment of patients. However, the traditional assessment method relies on the subjective observation of doctors, and there are problems such as non-uniform evaluation standard, insufficient repeatability, and difficulty in real-time feedback.

[0003] In recent years, researchers have proposed various automated assessment schemes to improve objectivity and efficiency, such as video assessment methods based on deep learning, but these methods mostly focus on single modal input and cannot cover comprehensive movement information in real scenes, and have insufficient robustness to abnormal actions. Specifically, only using video action detection methods, the score is severely distorted in environments such as occlusion and low light; specifically, a single modal inertial measurement unit implemented through an IMU sensor can record joint local acceleration and angular velocity and other movement parameters, but due to the lack of spatial reference, it is difficult to independently complete the accurate reconstruction of three-dimensional motion state, limiting its application in posture assessment. In addition, in terms of multi-modal, current technologies are difficult to effectively integrate and model the confidence of data from multiple modalities, lack reliable multi-modal fusion and uncertainty evaluation mechanism, limiting its applicability in clinically reliable scenarios. SUMMARY

[0004] The purpose of the present application is to provide a stroke patient upper limb movement scoring system, method and device for bed assessment scene, effectively fusing visual and inertial motion data, overcoming the limitations of single modal, and realizing accurate assessment of upper limb movement function in bed scene.

[0005] To achieve the above purpose, the present application provides the following solutions:

[0006] In a first aspect, the present application provides a stroke patient upper limb movement scoring system for bed assessment scene, comprising:

[0007] A multi-modal data acquisition module is configured to acquire video optical flow data and inertial motion data during the movement of the upper limbs of the stroke patient in the bed assessment scene;

[0008] A feature extraction module is configured to extract features from the video optical flow data and the inertial motion data using a neural network to obtain a visual modal representation vector and a motion modal representation vector;

[0009] a feature fusion module configured to perform cross-attention calculation on the visual modality representation vector and the action modality representation vector by using a CrossAttention attention mechanism to obtain fused visual features and fused action features;

[0010] a score and uncertainty prediction module configured to determine an upper limb movement score value and a corresponding confidence interval according to the fused visual features and the fused action features by combining an Evidential Deep Learning framework and a Dempster-Shafer evidence fusion theory.

[0011] In a second aspect, the present application provides a stroke patient upper limb movement score method for a bedridden assessment scenario, which applies the stroke patient upper limb movement score system for the bedridden assessment scenario, and the method comprises the following steps:

[0012] acquiring video optical flow data and inertial motion data in the process of upper limb movement of a stroke patient in a bedridden assessment scenario by using a multi-modal data acquisition module;

[0013] extracting features from the video optical flow data and the inertial motion data by using a neural network to obtain a visual modality representation vector and an action modality representation vector through a feature extraction module;

[0014] performing cross-attention calculation on the visual modality representation vector and the action modality representation vector by using a CrossAttention attention mechanism to obtain fused visual features and fused action features through a feature fusion module;

[0015] determining an upper limb movement score value and a corresponding confidence interval according to the fused visual features and the fused action features by using a score and uncertainty prediction module in combination with an Evidential Deep Learning framework and a Dempster-Shafer evidence fusion theory.

[0016] In a third aspect, the present application provides a computer device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement a stroke patient upper limb movement score method for a bedridden assessment scenario.

[0017] According to the specific embodiments provided in this application, the present application has the following technical effects: This application collects video optical flow data and inertial motion data from stroke patients during upper limb activity in a bedridden assessment scenario, which serves as the data foundation for all subsequent processing. This effectively addresses the common limb occlusion and illumination changes in bedridden scenarios, enhancing the robustness of action recognition to occlusion and posture variations in bedridden patients. Furthermore, by introducing inertial motion data, the shortcomings of single visual information in spatial motion perception are overcome. In particular, the combination of visual and inertial information enables a more comprehensive motion state representation framework. This application also uses the Cross Attention mechanism for multimodal feature interaction processing, combining the Evidential Deep Learning framework with the Dempster-Shafer evidence fusion theory to calculate upper limb motion scores and confidence intervals, achieving robust prediction and confidence output of upper limb motion assessment results, significantly improving the objectivity, automation level, and clinical practicality of the assessment system. In summary, this application not only improves the objectivity and automation level of the NIHSS scoring process, but also provides scalable core technical support for the engineering implementation of intelligent stroke rehabilitation assessment systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 This is a structural diagram of an upper limb movement scoring system for stroke patients in a bedridden assessment scenario in one embodiment of the present application.

[0020] Figure 2 It is the acquisition scene graph of the video acquisition submodule.

[0021] Figure 3 Schematic diagram of the data processing steps in the scoring and uncertainty prediction module.

[0022] Figure 4 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0024] This application addresses the problems of insufficient accuracy, strong subjective dependence and limited automation in the clinical assessment of upper limb motor function in bedridden stroke patients (NIHSS score). An assessment method based on multimodal fusion of video optical flow and wrist IMU signals is proposed.

[0025] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0026] In an exemplary embodiment, Figure 1 As shown, a stroke patient upper limb movement scoring system for bedridden assessment scenarios is provided, including: a multimodal data acquisition module, a feature extraction module, a feature fusion module, and a scoring and uncertainty prediction module.

[0027] (1) The multimodal data acquisition module is used to collect video optical flow data and inertial motion data of stroke patients in bed assessment scenarios and during upper limb activities; specifically, the multimodal data acquisition module includes a video acquisition submodule, an inertial motion acquisition submodule, an optical flow generation submodule, and an inertial preprocessing submodule.

[0028] The video acquisition submodule is used to collect RGB video sequences of stroke patients performing preset upper limb movements in the bedside assessment scene. Considering that stroke patients are often bedridden or have difficulty in moving, the design is specially adapted to the bedside assessment scene, so that the assessment process can be completed while the patient is in bed. Figure 2 As shown in the figure, the video acquisition submodule includes a workbench and a video acquisition device. The video acquisition device is an integrated depth camera (such as Orbbec Femto Bolt). The video acquisition device is placed on the workbench on the side of the bed and is used to capture RGB video sequences of the patient performing specified upper limb movements (such as extending the arm, clenching the fist, raising the arm, etc.).

[0029] The inertial motion acquisition submodule collects triaxial acceleration and angular velocity signals from stroke patients performing pre-defined upper limb movements within a bedridden assessment scenario. In practical applications, a wearable inertial measurement unit (IMU) is deployed to synchronously collect multi-channel motion data from the patient's upper limbs. Each IMU integrates a triaxial accelerometer and a triaxial gyroscope, with a sampling frequency of 50Hz or higher, ensuring high-precision capture of subtle dynamic changes in movement.

[0030] In the actual deployment process, the IMU module is fixed to the key positions of the patient's forearm and upper arm through flexible straps to adapt to the evaluation needs in the bedridden state. Then the three-axis acceleration and angular velocity signals generated by the upper limbs of the stroke patients during the action process are synchronously acquired. After the collection is completed, data communication is carried out with the upper computer through Bluetooth or Wi-Fi to ensure real-time and continuous data uploading. The collected three-axis acceleration and angular velocity signals contain six channels of acceleration (Ax, Ay, Az) and angular velocity (Gx, Gy, Gz), thereby forming a multi-dimensional time series.

[0031] Obviously, the video acquisition submodule and the inertial motion acquisition submodule are responsible for acquiring the multi-modal raw data of the patient's upper limb activity process, i.e. the RGB video sequence and the inertial motion information. Then, the multi-modal raw data is synchronously acquired and uploaded to the upper computer placed at the workbench through Bluetooth or a serial port, and enters the optical flow generation submodule and the inertial preprocessing submodule for subsequent preprocessing and feature analysis processes.

[0032] The optical flow generation submodule is connected with the video acquisition submodule, and the optical flow generation submodule is used for generating video optical flow data according to the RGB video sequence; specifically, the following data processing steps are included: using a TV-L1 optical flow estimation algorithm to estimate the motion vector of each pixel in the horizontal direction and the vertical direction in all adjacent video frames in the RGB video sequence to generate an optical flow map sequence; the optical flow map sequence is the video optical flow data.

[0033] Among them, the TV-L1 optical flow estimation algorithm is an optical flow estimation method based on variational method, which has high robustness and strong edge preservation ability, and is particularly suitable for the clinical scene of relatively static background and small action amplitude of the patient in the bedridden state. In the above application, adjacent video frames are used as input, and the motion vector of each pixel in the horizontal direction and the vertical direction is estimated respectively to generate a pair of two-dimensional optical flow maps (U component and V component). By processing all continuous frame pairs, the system forms a complete optical flow map sequence. The optical flow sequence can be further stacked into a three-dimensional tensor structure as the input of the subsequent feature extraction module.

[0034] The calculation of the optical flow map sequence can be realized in the local upper computer or the edge processing unit, which supports real-time conversion and buffering of the RGB video sequence, and ensures that the entire evaluation process can quickly start the feature extraction and scoring link after the collection is completed.

[0035] The inertial pretreatment submodule is connected with the inertial motion acquisition submodule, and is used for pretreating the triaxial acceleration and angular velocity signals to obtain inertial motion data with unified format. In order to improve the generalization ability of the model and the data quality, the pretreatment includes time alignment processing, denoising processing, normalization processing, abnormality detection processing, and resampling and clipping processing. The time alignment processing is to align the triaxial acceleration and angular velocity signals with the video optical flow data in the optical flow generation submodule on the same time axis, so as to support subsequent multi-modal fusion. The denoising processing is to remove high-frequency noise by using a low-pass filter (such as a Butterworth filter). The normalization processing is to perform z-score standardization on each channel to improve the comparability between different individuals. The abnormality detection is to automatically identify stationary drift or sudden outlier signals and prompt reacquisition. The resampling and clipping are to unify the signal length and clip as needed to a fixed time window (such as 6 seconds).

[0036] The inertial motion data with unified format obtained after pretreatment is sent to a feature extraction module to extract local multi-scale motion features thereof, so as to provide high-quality input for subsequent fusion calculation with visual information.

[0037] (II) A feature extraction module is used to extract features of the video optical flow data and the inertial motion data by using a neural network, so as to obtain a visual modal representation vector and a motion modal representation vector, and provide semantic support for subsequent fusion and scoring. The feature extraction module includes the following data processing steps:

[0038] (1) Obtain a trained ResNet three-dimensional convolutional neural network and an InceptionTime neural network. In actual application, the training process of the ResNet three-dimensional convolutional neural network includes: training ResNet3D-50 by using a first training sample set; each training sample in the first training sample set includes a sequence of optical flow maps generated by a TV-L1 algorithm from each RGB video, and a spatiotemporal motion feature as a label. The optical flow map reflects the direction and amplitude of the pixel motion between consecutive frames. The training process of the InceptionTime neural network includes: training InceptionTime by using a second training sample set; each training sample in the second training sample set includes an IMU multi-channel time sequence and a motion modal representation vector as a label.

[0039] (2) encode each optical flow map sequence in the video optical flow data into a fixed frame length video tensor, and then input into the ResNet three-dimensional convolutional neural network to obtain a visual modal representation vector; wherein the ResNet three-dimensional convolutional neural network can model local spatial structure and temporal dynamics at the same time, extract deep video representation through 3D convolutional layers layer by layer, and then input into the Transformer module to capture long-range dependencies between frames in the video through self-attention mechanism, and output the visual modal representation vector

[0040] (4) input the inertial motion data into the InceptionTime neural network to obtain an action modal representation vector. In actual application, the inertial motion data can be standardized first, and then input into the InceptionTime neural network. The InceptionTime neural network adopts a multi-scale one-dimensional convolution module to capture local patterns under different time receptive fields, and uses a residual structure to improve feature extraction efficiency and stability. Finally, a global feature vector is output, representing the action performance under the IMU mode, i.e. obtaining the action modal representation vector

[0041] The feature extraction module finally outputs two modal representation vectors with consistent dimensions, representing the embedding features of the visual modal and the action modal, respectively, providing input basis for the subsequent feature fusion module.

[0042] (Three) a feature fusion module for cross-computing attention on the visual modal representation vector and the action modal representation vector by using a Cross Attention attention mechanism to obtain fused visual features and fused action features. In order to realize deep coupling between multi-modal information, the present application proposes a Cross Attention-based modal alignment and fusion mechanism, which calculates attention on two different sequences to process the semantic relationship between the two sequences, so as to better fuse multi-modal information. The feature fusion module includes the following data processing steps:

[0043] When the visual modal representation vector is used as input and the action modal representation vector is used as target, the cross attention attention mechanism is used to calculate the input and query to obtain the fused visual features; when the action modal representation vector is used as input and the visual modal representation vector is used as target, the cross attention attention mechanism is used to calculate the input and query to obtain the fused action features.

[0044] Wherein, the calculation process of the fused action features comprises: projecting the input and the target into query Q=w q Xvis , key K = w k X imu , value V = w v X imu ; Among them, w q 、w k 、w v Both are learnable parameter matrices, X vis is the target visual modality representation vector, X imu is the action modality representation vector as input;

[0045] The following function is used to represent the fusion action features:

[0046] Among them, d k is the dimension of the key vector, used as a scaling factor to avoid gradient vanishing or exploding problems caused by excessive dimensions; () T Indicates transpose.

[0047] (IV) Scoring and uncertainty prediction module, which is used to combine the Evidential Deep Learning framework with the Dempster-Shafer evidence fusion theory to determine the upper limb movement score value and the corresponding confidence interval based on the fused visual features and the fused motion features. Figure 3 As shown, the scoring and uncertainty prediction module includes the following data processing steps:

[0048] (1) Based on the Evidential Deep Learning framework, confidence modeling is performed for the visual modality and the action modality respectively to obtain the visual EDL scoring model and the action EDL scoring model.

[0049] (2) Inputting the fused visual features into the visual EDL scoring model to obtain visual scoring evidence, and then generating a visual prediction distribution based on the visual scoring evidence according to the Dirichlet distribution hypothesis.

[0050] (3) Inputting the fused action features into the action EDL scoring model to obtain action scoring evidence, and then generating an action prediction distribution based on the action scoring evidence according to the Dirichlet distribution hypothesis.

[0051] The model built by the Evidential Deep Learning framework does not directly output the category probability, but outputs the score evidence amount e = [e1, e2, ..., e K]According to the Dirichlet distribution assumption, the model constructs a hyperparameter vector α = e + 1, thereby forming a predictive distribution rather than a single-point probability. This structure can effectively express the model's confidence in each category and introduce a credibility mechanism for the scoring task.

[0052] (4) The visual prediction distribution and the action prediction distribution are respectively mapped into a visual basic probability assignment and an action basic probability assignment, and then based on the Dempster-Shafer fusion rule, joint modeling under conflict awareness is performed to obtain a fused evidence according to the visual basic probability assignment and the action basic probability assignment.

[0053] In order to integrate the complementarity of the visual modality (video optical flow sequence) and the motion modality (IMU signal) at the evidence level, the Dempster-Shafer evidence fusion theory is further introduced. The evidence output of each modality is mapped into a basic probability assignment (BBA), that is:

[0054]

[0055] Wherein, m ∈ {vis, imu}, vis corresponds to the visual modality, which can also be represented by v; imu corresponds to the action modality, which can also be represented by i; b k is the kth basic probability, e j is the jth visual scoring evidence quantity. Based on the Dempster fusion rule, the evidences of the two modalities are jointly modeled under conflict awareness:

[0056]

[0057] Wherein, is the fused evidence.

[0058] (5) Based on the fused evidence, the Dirichlet distribution parameters are reconstructed, and the upper limb motion score value and the corresponding confidence interval are generated.

[0059] At this point, the scoring and uncertainty prediction module not only outputs specific score results, but also calculates the expectation, variance and uncertainty entropy of the score distribution, and further constructs the score confidence interval. Especially when the model uncertainty is high, the system can automatically prompt the doctor to review, ensuring that the prediction result is safe and controllable, and interpretable.

[0060] In summary, the system of the present application, facing the NIHSS motion score task, realizes multi-modal neural function evaluation, innovatively fuses visual and inertial modal information, constructs an end-to-end scoring and credible output mechanism, has strong robustness and credibility, and is particularly suitable for clinical application scenarios such as remote rehabilitation evaluation and bedside monitoring. In summary, the system has high accuracy, strong generalization and good scalability through the cooperative work of the above modules, and realizes the objective and automatic evaluation of the upper limb function state of the patient in the bed scene.

[0061] Based on the same inventive concept, the embodiments of the present application also provide a stroke patient upper limb motion scoring method for a bed evaluation scene, which applies the stroke patient upper limb motion scoring system for a bed evaluation scene described above. The method comprises:

[0062] Step 100, using a multi-modal data acquisition module to collect video optical flow data and inertial motion data during the movement of the upper limbs of the stroke patient in the bed evaluation scene.

[0063] Step 200, using a feature extraction module to extract features from the video optical flow data and the inertial motion data respectively by using a neural network, to obtain a visual modal representation vector and a motion modal representation vector.

[0064] Step 300, using a CrossAttention attention mechanism to cross-calculate attention on the visual modal representation vector and the motion modal representation vector by using a feature fusion module, to obtain fused visual features and fused motion features.

[0065] Step 400, using a scoring and uncertainty prediction module to combine an Evidential Deep Learning framework and a Dempster-Shafer evidence fusion theory, to determine an upper limb motion score value and a corresponding confidence interval according to the fused visual features and the fused motion features.

[0066] Compared with the prior art, the present application has the following advantages:

[0067] (1) Video and IMU cooperative acquisition and preprocessing mechanism, suitable for bed patient motion scene, ensuring signal integrity and time synchronization accuracy.

[0068] (2) Modal feature extraction and interaction structure design: optical flow map-ResNet3D and IMU-InceptionTime are used to extract intra-modal motion information, and Cross-Attention is used to realize explicit inter-modal dependency modeling, improve feature fusion capability, and effectively capture cooperative motion patterns.

[0069] (3) Uncertainty modeling mechanism based on Evidential Deep Learning: confidence modeling is performed on each modality output respectively, Dempster-Shafer evidence fusion is introduced to weight and regulate the conflicting information between modalities, and finally the scoring output with confidence interval is realized. The above processing solves the problem of conflict between modalities and guarantees the stability of prediction.

[0070] (4) Automatic scoring module and result visualization feedback: the system can output the score value and its confidence, has interpretability, realizes auxiliary decision support and clinical credible evaluation.

[0071] (5) Not dependent on high-quality video or multiple sensors, only single camera + low-cost IMU is needed, suitable for bedside assessment; strong scalability, suitable for telemedicine and intelligent rehabilitation auxiliary system.

[0072] (6) The whole application adopts an end-to-end structure design, fuses video information and inertial measurement unit (IMU) signals to improve the accuracy and robustness of scoring.

[0073] In an exemplary embodiment, a computer device, which can be a server or a terminal, has an internal structure diagram as shown in Figure 4 The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a stroke patient upper limb motor score method for bed assessment scenarios.

[0074] Those skilled in the art can understand that Figure 4 the structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0075] In an example embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the steps of the above method embodiments.

[0076] In an example embodiment, a computer readable storage medium is provided, storing a computer program which, when executed by a processor, implements the steps of the above method embodiments.

[0077] In an example embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the above method embodiments.

[0078] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0079] It can be understood by those skilled in the art that all or part of the processes in the above method embodiments can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above method embodiments. Any reference to memory, database or other medium used in the embodiments provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0080] The database involved in each embodiment provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, and the like, without being limited thereto. The processor involved in each embodiment provided by the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, and the like, without being limited thereto.

[0081] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, but it should be considered that any combination of the technical features is within the scope of the present disclosure, as long as the combination does not result in contradictions.

[0082] The principles and implementation modes of the present application are described by applying specific examples herein, and the above embodiments are only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, the specific implementation modes and application ranges will be changed according to the idea of the present application. In summary, the content of the present description should not be understood as a limitation of the present application.

Claims

1. A stroke patient upper limb movement scoring system for bedridden assessment scenarios, characterized by: The system comprises: Multimodal data acquisition module, used to collect video optical flow data and inertial motion data of stroke patients during upper limb activities in bed assessment scenarios; a feature extraction module, configured to extract features from the video optical flow data and the inertial motion data using a neural network to obtain a visual modality representation vector and a motion modality representation vector; A feature fusion module is used to use the CrossAttention mechanism to perform cross-calculation attention on the visual modality representation vector and the action modality representation vector to obtain fused visual features and fused action features; The scoring and uncertainty prediction module is used to combine the Evidential Deep Learning framework with the Dempster-Shafer evidence fusion theory to determine the upper limb movement score value and the corresponding confidence interval based on the fused visual features and the fused motion features.

2. The upper limb movement scoring system for stroke patients in bed resting assessment scenarios according to claim 1 is characterized in that: The multimodal data acquisition module includes a video acquisition submodule, an inertial motion acquisition submodule, an optical flow generation submodule and an inertial preprocessing submodule; The video acquisition submodule is used to acquire RGB video sequences of stroke patients performing preset upper limb movements in a bed rest assessment scenario; The inertial motion acquisition submodule is used to collect the three-axis acceleration and angular velocity signals of the stroke patient performing the preset upper limb movements in the bed assessment scenario; The optical flow generation submodule is connected to the video acquisition submodule, and the optical flow generation submodule is used to generate video optical flow data according to the RGB video sequence; The inertial preprocessing submodule is connected to the inertial motion acquisition submodule, and the inertial preprocessing submodule is used to preprocess the three-axis acceleration and angular velocity signals to obtain inertial motion data in a unified format.

3. The upper limb movement scoring system for stroke patients in bed resting assessment scenarios according to claim 2, characterized in that: The optical flow generation submodule includes the following data processing steps: The TV-L1 optical flow estimation algorithm is used to estimate the motion vector of each pixel in the horizontal and vertical directions for all adjacent video frames in the RGB video sequence to generate an optical flow map sequence; the optical flow map sequence is video optical flow data.

4. The upper limb movement scoring system for stroke patients in bed resting assessment scenarios according to claim 2, characterized in that: The preprocessing in the inertial preprocessing submodule includes time alignment, denoising, normalization, anomaly detection, and resampling and cropping. The time alignment is to align the three-axis acceleration and angular velocity signals with the video optical flow data in the optical flow generation submodule on the same time axis.

5. The upper limb movement scoring system for stroke patients in bed rest assessment scenarios according to claim 1, characterized in that: The feature extraction module includes the following data processing steps: Obtain the trained ResNet 3D convolutional neural network and InceptionTime neural network; Encoding each optical flow graph sequence in the video optical flow data into a video tensor of fixed frame length, and then inputting the video tensor into the ResNet three-dimensional convolutional neural network to obtain a visual modality representation vector; The inertial motion data is input into the InceptionTime neural network to obtain a motion modality representation vector.

6. The upper limb movement scoring system for stroke patients in bed resting assessment scenarios according to claim 1, characterized in that: The feature fusion module includes the following data processing steps: When the visual modality representation vector is used as input and the action modality representation vector is used as target, the crossattention mechanism is used to calculate the input and query to obtain a fused visual feature; When the action modality representation vector is used as input and the visual modality representation vector is used as target, the crossattention mechanism is used to calculate the input and query to obtain the fused action feature; The calculation process of fusion action features includes: projecting the input and target into: query Q = w q X vis , key K = w k X imu , value V = w v X imu ; Among them, w q 、w k 、w v Both are learnable parameter matrices, X vis is the target visual modality representation vector, X imu is the action modality representation vector as input; The following function is used to represent the fusion action features: Among them, d k is the dimension of the key vector, () T Indicates transpose.

7. The upper limb movement scoring system for stroke patients in bed rest assessment scenarios according to claim 1, characterized in that: The scoring and uncertainty prediction module includes the following data processing steps: Based on the Evidential Deep Learning framework, confidence modeling is performed for the visual modality and the action modality to obtain the visual EDL scoring model and the action EDL scoring model respectively; Inputting the fused visual features into the visual EDL scoring model to obtain visual scoring evidence, and then generating a visual prediction distribution based on the visual scoring evidence according to the Dirichlet distribution hypothesis; Inputting the fused action features into the action EDL scoring model to obtain action score evidence, and then generating an action prediction distribution based on the action score evidence according to the Dirichlet distribution hypothesis; The visual prediction distribution and the action prediction distribution are mapped into a visual basic probability distribution and an action basic probability distribution, respectively, and then, based on the Dempster-Shafer fusion rule, joint modeling under conflict perception is performed to obtain fused evidence according to the visual basic probability distribution and the action basic probability distribution; Based on the fused evidence, the Dirichlet distribution parameters are reconstructed, and upper limb movement score values ​​and corresponding confidence intervals are generated.

8. The upper limb movement scoring system for stroke patients in bed rest assessment scenarios according to claim 7, characterized in that: The function formula of basic probability distribution is: Among them, m∈{vis,imu}, vis corresponds to the visual mode, which can also be represented by v; imu corresponds to the action mode, which can also be represented by i; b k is the kth basic probability, e j The amount of evidence for the jth visual score; The functional formula for joint modeling is: in, Evidence for integration.

9. A method for scoring upper limb movement of stroke patients in bed-ridden assessment scenarios, using the upper limb movement scoring system for stroke patients in bed-ridden assessment scenarios according to any one of claims 1 to 8, characterized in that: The method comprises: A multimodal data acquisition module was used to collect video optical flow data and inertial motion data of stroke patients during bed rest assessment and upper limb activity. Using a neural network to extract features from the video optical flow data and the inertial motion data respectively through a feature extraction module to obtain a visual modality representation vector and a motion modality representation vector; A feature fusion module adopts a CrossAttention mechanism to perform cross-calculation attention on the visual modality representation vector and the action modality representation vector to obtain a fused visual feature and a fused action feature; The scoring and uncertainty prediction module is combined with the Evidential Deep Learning framework and the Dempster-Shafer evidence fusion theory to determine the upper limb movement score value and the corresponding confidence interval based on the fused visual features and the fused motion features.

10. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the upper limb movement scoring method for stroke patients in bed resting assessment scenarios as described in claim 9.

Images