Patient dyskinesia assessment system and method based on multi-modal data fusion

Abstract

The embodiment of the invention provides a patient dyskinesia assessment system and method based on multi-modal data fusion, and relates to the field of motion function assessment and artificial intelligence. The system comprises an acquisition module used for acquiring multi-modal motion performance data of a to-be-evaluated target; the first extraction module is used for extracting human body bounding boxes and segmentation masks of the image data and the video data; the enhancement module is used for extracting image features of the human body bounding box and the segmentation mask and enhancing the image features; the second extraction module is used for extracting signal features of the preprocessed physiological signal data and voice signal data based on a deep neural network and enhancing the signal features to obtain text features; the updating module is used for performing feature updating; the fusion module is used for carrying out feature fusion on the updated image features and text features; and the evaluation module is used for obtaining a motion function evaluation result of the to-be-evaluated target. According to the scheme, the accuracy and efficiency of the dyskinesia assessment result of the patient are effectively improved.

Description

Technical Field

[0001] The present disclosure relates to the fields of motor function assessment and artificial intelligence technology, and in particular to a system and method for assessing patient motor dysfunction based on multimodal data fusion. Background Art

[0002] Traditional motor function assessment methods rely primarily on manual observation and subjective judgment, which are subject to significant subjectivity and uncertainty. In recent years, with the rapid development of artificial intelligence and deep learning technologies, motor function assessment has garnered widespread attention in the medical field. However, while traditional motor function assessment systems provide rich information through multimodal data, data acquisition and processing techniques for different modalities may differ, resulting in suboptimal data fusion and complementarity. Furthermore, existing assessment models may not fully adapt to complex motor function assessment needs, leading to inaccurate assessment results. Factors such as model training data and algorithm design can affect assessment accuracy. Furthermore, multimodal data preprocessing, feature extraction, and fusion are often complex, requiring significant computational resources and time. This can result in poor real-time performance, making the assessment system incapable of meeting certain urgent or high-frequency assessment needs. Therefore, traditional motor function assessment systems based on multimodal data still have some shortcomings in terms of accuracy and efficiency. Summary of the Invention

[0003] To address the shortcomings of existing technologies, the present disclosure provides a system and method for assessing patient motor dysfunction based on multimodal data fusion. This addresses the technical issues that traditional motor function assessment methods rely primarily on manual observation and subjective judgment, which are subject to significant subjectivity and uncertainty. Furthermore, existing motor function assessment systems based on multimodal data often lack accuracy and efficiency in their assessment results.

[0004] In a first aspect, an embodiment of the present disclosure provides a patient movement dysfunction assessment system based on multimodal data fusion, comprising:

[0005] an acquisition module, configured to acquire multimodal motion performance data of a target to be evaluated and preprocess the multimodal motion performance data based on a preprocessing layer; the multimodal motion performance data includes physiological signal data, voice signal data, image data, and video data;

[0006] A first extraction module is used to extract the human body bounding box and segmentation mask of the preprocessed image data and video data based on the mask residual network;

[0007] An enhancement module, configured to extract image features of the human body bounding box and the segmentation mask based on a multi-layer convolutional structure, and enhance the image features based on reinforcement learning to obtain enhanced image features;

[0008] A second extraction module is configured to extract signal features from the preprocessed physiological signal data and speech signal data based on a deep neural network, and enhance the signal features based on reinforcement learning to obtain text features;

[0009] An updating module, configured to update the enhanced image features and the text features based on the attention mechanism module to obtain updated image features and updated text features;

[0010] A fusion module, configured to fuse the updated image features and the updated text features to obtain fused feature data;

[0011] The evaluation module is used to input the fused feature data into the fully connected layer to obtain the motor function evaluation result of the target to be evaluated.

[0012] In one possible implementation, the preprocessing of the multimodal motion performance data based on the preprocessing layer includes: performing data enhancement, data cleaning and data normalization operations on the multimodal motion performance data.

[0013] In one possible implementation, the first extraction module is configured to extract a human body bounding box and a segmentation mask from the preprocessed image data and video data based on a mask residual network, comprising:

[0014] The preprocessed image data and video data are passed through the preprocessing layer. The bounding box of the human body is first detected in the input image through the mask residual network. Then, the classification and bounding box regression tasks are combined and an additional branch is used to generate the segmentation mask of the object.

[0015] In one possible implementation, the total loss TotalLoss of the masked residual network is: TotalLoss = λ c ×ClassLoss+λ r ×BoxLoss+λ m ×MaskLoss+λ n ×GioULoss;

[0016] in,

[0017]

[0018] in,

[0019]

[0020] Among them, TotalLoss is the overall loss, MaskLoss is the segmentation mask loss, m ij is the ground-truth segmentation mask for the jth pixel of the i-th bounding box, is the predicted segmentation mask probability value; BoxLoss is the bounding box regression loss, ti k is the true value of the k-th coordinate (x, y, w, h) of the i-th bounding box, is the predicted value; ClassLoss is the classification loss, y ij is the true label of the jth category of the i-th bounding box, is the predicted probability; λ c ,λ r ,λ m ,λ n is a hyperparameter, b x and b y are the width and height of the predicted bounding box, A pred and A true are the areas of the predicted bounding box and the true bounding box, respectively, and IoU is the ratio of the intersection area of the predicted bounding box and the true bounding box to the union area.

[0021] In one possible implementation, the extracting of image features of the human body bounding box and segmentation mask based on a multi-layer convolutional structure includes:

[0022] Mask_ResNet(x)=ReLU(conv1(x)+skip1)+conv2(ReLU(conv1(x)+skip1))

[0023] Among them, Mask_ResNet(x) means inputting the input data x into the multi-layer convolution structure, ReLU is the activation function, conV l represents the convolution operation of the lth layer, x represents the input data, and skip1 represents the skip connection output of the first residual block.

[0024] In one possible implementation, enhancing the image features based on reinforcement learning includes enhancing the image features based on a Q-value function, and the specific process is as follows:

[0025] Q(s t , a t )←Q(s t , a t )+α[R t+1 +ymaxQ(S t+1 ,a′)-Q(s t , a t )]

[0026] Where: Q(st , a t ) is the state s at time step t t Take action a t Q value; R t+1 is the immediate reward obtained at time step t+1; γ is the discount factor used to consider the discount of future returns; maxQ(s t+1 , a′) is in the next state S t+1 The maximum Q value of all possible actions a′ under α is the learning rate, which controls the size of the update.

[0027] In one possible implementation, the deep neural network includes three hidden layers, namely L1, L2, and L3, and each hidden layer has a corresponding weight matrix W l and the bias vector b l , the activation function is ReLU;

[0028] The forward propagation process of the deep neural network is expressed as:

[0029] X l+1 =σ(W l X l +b l )

[0030] Among them, X0 is the input physiological signal data, X l+1 is the output of the lth layer, and σ(.) is the ReLU activation function.

[0031] In one possible implementation, the updated feature data X′ is calculated using the following formula:

[0032]

[0033] Among them, the attention weight a j Through the softmax function, we get: j =softmax(A[:,j]), and a j ∈R 1×N ; N represents the total number of image blocks in the image feature data set; x′ j Represents the j-th image feature data x j The corresponding vector; V represents the value matrix of the self-attention mechanism and V∈R N×D′ ; D′ is the dimension after flattening; D″ is the dimension of positional encoding; ⊙ represents element-wise multiplication;

[0034] The fused feature data F is expressed as:

[0035] F=αF X +(1-α)F T

[0036] Among them, F is the feature data after fusion, F x is the image feature after the update, FT is the text feature after the update, and α is the weight parameter.

[0037] In one possible implementation, the evaluation module is configured to input the fused feature data into a fully connected layer to obtain a motor function evaluation result of the target to be evaluated, including:

[0038] The output F of the feature fusion module is converted into a probability distribution through the softmax function to predict different patterns of patient movement disorders. The specific conversion algorithm used is as follows:

[0039] P(y|X)=softmax(F)

[0040] Where y is the patient's movement disorder category, and P(y|X) is the probability of outputting category y given the input data X.

[0041] In a second aspect, the present disclosure provides a method for assessing patient motor dysfunction based on multimodal data fusion, comprising:

[0042] Acquire multimodal motion performance data of the target to be evaluated, and preprocess the multimodal motion performance data based on the preprocessing layer; the multimodal motion performance data includes physiological signal data, voice signal data, image data, and video data;

[0043] Based on the mask residual network, the human body bounding box and segmentation mask are extracted from the preprocessed image data and video data;

[0044] Extracting image features of the human body bounding box and the segmentation mask based on a multi-layer convolutional structure, and enhancing the image features based on reinforcement learning to obtain enhanced image features;

[0045] Extracting signal features from pre-processed physiological signal data and speech signal data based on a deep neural network, and enhancing the signal features based on reinforcement learning to obtain text features;

[0046] Based on the attention mechanism module, feature updating is performed on the enhanced image features and the text features respectively to obtain updated image features and updated text features;

[0047] Performing feature fusion on the updated image features and the updated text features to obtain fused feature data;

[0048] The fused feature data is input into the fully connected layer to obtain the motor function evaluation result of the target to be evaluated.

[0049] The present invention provides a system and method for assessing patient motor dysfunction based on multimodal data fusion, which utilizes multimodal data fusion technology to capture and analyze the patient's motor performance in real time, and identifies the patterns and characteristics of motor disorders through deep learning algorithms. The cross-attention mechanism enhances the complementarity between different modal data, improves the effect of feature extraction and fusion, and makes motor function assessment more accurate and efficient. This overcomes the technical problems that traditional motor function assessment methods mainly rely on manual observation and subjective judgment, which are subject to great subjectivity and uncertainty, and the existing motor function assessment systems based on multimodal data are insufficient in terms of the accuracy and efficiency of the assessment results. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 A schematic diagram of the units of a patient movement dysfunction assessment system based on multimodal data fusion provided in an embodiment of the present disclosure.

[0052] Figure 2 A technical schematic diagram of a patient movement dysfunction assessment system based on multimodal data fusion provided by an embodiment of the present disclosure;

[0053] Figure 3 A flowchart of a method for assessing patient motor dysfunction based on multimodal data fusion provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0054] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure.

[0055] Those skilled in the art will understand that the terms "first", "second" and the like in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and do not represent any specific technical meaning, nor do they represent the necessary logical order between them. It should also be understood that in the embodiments of the present disclosure, "multiple" may refer to two or more, and "at least one" may refer to one, two or more. It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly defined or given a contrary revelation in the context. In addition, the term "and / or" in the present disclosure is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship. It should also be understood that the description of each embodiment in the present disclosure emphasizes the differences between the embodiments, and the same or similar aspects thereof can be referenced to each other. For the sake of brevity, they will not be described one by one.

[0056] At the same time, it should be understood that for ease of description, the dimensions of the various parts shown in the drawings are not drawn according to the actual proportional relationship. The following description of at least one exemplary embodiment is actually only illustrative and is in no way intended to limit the present disclosure and its application or use. Technologies, methods and equipment known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods and equipment should be considered part of the specification. It should be noted that similar numbers and letters represent similar items in the following figures, so once an item is defined in one figure, it does not need to be further discussed in subsequent figures.

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0058] The present disclosure proposes a multimodal motor function assessment algorithm based on a cross-attention mechanism and deep learning. The algorithm uses multimodal data fusion technology to capture and analyze the patient's motor performance in real time, and identifies the patterns and characteristics of movement disorders through a deep learning algorithm. Specifically, an improved multimodal fusion neural network is used to automatically analyze the captured data, thereby providing patients with accurate motor function assessments and assisting doctors in diagnosing diseases. This algorithm enhances the complementarity between different modal data through a cross-attention mechanism, improves the effect of feature extraction and fusion, and makes motor function assessment more accurate and efficient. Specifically, by capturing and analyzing the patient's multimodal data (such as video, audio, physiological signals, etc.), an automated assessment of motor function is achieved. The use of multimodal data fusion technology combined with a deep learning algorithm can not only capture the patient's motor performance in real time, but also identify the patterns and characteristics of movement disorders through automated analysis, providing doctors with a more accurate diagnostic basis, and greatly improving the accuracy and efficiency of the assessment.

[0059] Figure 1 1 is a module diagram of a patient movement dysfunction assessment system based on multimodal data fusion according to an embodiment of the present disclosure; Figure 2 This is a technical schematic diagram of a system and method for evaluating patient motor dysfunction based on multimodal data fusion provided by an embodiment of the present disclosure. Figure 1 and Figure 2 As shown, the system includes: an acquisition module 10, a first extraction module 20, an enhancement module 30, a second extraction module 40, an update module 50, a fusion module 60 and an evaluation module 70.

[0060] Specifically, the acquisition module 10 is used to obtain multimodal motion performance data of the target to be evaluated, and preprocess the multimodal motion performance data based on the preprocessing layer; the multimodal motion performance data includes physiological signal data, voice signal data, image data and video data.

[0061] Specifically, video data can be input as image data by reading it frame by frame. The signal data and image data are each preprocessed through a preprocessing layer, and the preprocessing operations include data enhancement, data cleaning, and data normalization.

[0062] The first extraction module 20 is used to extract the human body bounding box and segmentation mask of the pre-processed image data and video data based on the mask residual network.

[0063] Specifically, the first extraction module 20 passes the preprocessed image data and video data through the preprocessing layer, first through the mask residual network (Mask ReSNet) detects the bounding box of a person in the input image, then combines the classification and bounding box regression tasks and uses an additional branch to generate the segmentation mask of the object.

[0064] The enhancement module 30 is used to extract image features of the human body bounding box and the segmentation mask based on a multi-layer convolution structure, and enhance the image features based on reinforcement learning to obtain enhanced image features.

[0065] The second extraction module 40 is used to extract signal features of the pre-processed physiological signal data and speech signal data based on a deep neural network, and enhance the signal features based on reinforcement learning to obtain text features.

[0066] The updating module 50 is used to update the enhanced image features and text features based on the attention mechanism module to obtain updated image features and updated text features.

[0067] The fusion module 60 is used to fuse the updated image features and the updated text features to obtain fused feature data.

[0068] The evaluation module 70 is used to input the fused feature data into the fully connected layer to obtain the motor function evaluation result of the target to be evaluated.

[0069] Furthermore, in some embodiments, the first extraction module 20 is configured to extract a human body bounding box and a segmentation mask from the preprocessed image data and video data based on a mask residual network, including:

[0070] Among them, the mask residual network (Mask The overall loss TotalLoss of ResNet) includes the weighted sum of segmentation mask loss, bounding box regression loss, classification loss and GioU loss:

[0071] Specifically,

[0072] TotalLoss=λ c ×ClassLoss+λ r ×BoxLoss+λ m ×MaskLoss+λ n ×GioULoss;

[0073] in,

[0074] Segmentation mask loss:

[0075] Bounding Box Regression Loss:

[0076] in,

[0077] Classification loss:

[0078] GioU losses:

[0079] Among them, TotalLoss is the overall loss, MaskLoss is the segmentation mask loss, m ij is the ground-truth segmentation mask (0 or 1) of the jth pixel of the i-th bounding box, is the predicted segmentation mask probability value; BoxLoss is the bounding box regression loss, t ik is the true value of the k-th coordinate (x, y, w, h) of the i-th bounding box, is the predicted value; ClassLoss is the classification loss, y ij is the true label (0 or 1) of the jth category of the i-th bounding box, is the predicted probability (calculated by the softmax layer), ∑ j y ij =1;λ c ,λ r ,λ m ,λ n is a hyperparameter, b x and b y are the width and height of the predicted bounding box, A pred and A true are the areas of the predicted bounding box and the true bounding box, respectively, and IoU is the ratio of the intersection area of the predicted bounding box and the true bounding box to the union area.

[0080] In an optional implementation provided by the embodiments of the present disclosure, extracting image features using a multi-layer convolutional structure includes:

[0081] ResNet(x)=ReLU(conv1(x)+skip1)+conv2(ReLU(conv1(x)+skip1))

[0082] Among them, ReLU is the activation function, conv l represents the convolution operation of layer l, x represents the input data, and skip1 represents the skip connection output of the first residual block. Skip connections are a common technique in deep neural networks to alleviate the problems of vanishing and exploding gradients during deep network training. The basic idea is to introduce direct connections into the network, allowing information in the network to propagate directly to deeper layers, thereby preserving more gradient information.

[0083] Image feature data X∈RH×W×D , where H is the height, W is the width, and D is the depth (number of channels). The present disclosure divides the image into multiple patches (patch), and the size of each patch is P×P, where P is a positive integer. In this way, the image feature data can be represented as a set of patches X = {x1, x2, ..., x N},in Is the total number of patches. Each patchx i is flattened into a vector x′ i ∈R D＇ , where D′=P×P×D is the dimension after flattening. Then, the present disclosure adds a position encoding vector p to each patch i ∈R D″ , where D' is the dimension of the positional encoding. Thus, the input vector of each patch becomes z i =[x′ i ;p i ].

[0084] In an optional implementation provided by the embodiment of the present disclosure, enhancing the image features based on reinforcement learning includes: enhancing the image features based on a Q-value function, and the specific process is as follows:

[0085] Specifically, the Q-value function can help the model select important features. In each state s, the model can use the Q-value function to evaluate the importance of different features. Features with higher Q-values are given higher weights, thus playing a greater role in the decision-making process.

[0086] The specific process of enhancing the image features based on the Q-value function is as follows:

[0087] Q(s t , a t )←Q(s t , a t )+α[R t+1 +YmaxQ(s t+1 ,a′)-Q(s t , a t )]

[0088] Where: Q(s t , a t ) is the state s at time step t t Take action a t Q value; R t+1 is the immediate reward obtained at time step t+1; γ is the discount factor used to consider the discount of future returns; maxQ(s t+1 , a′) is in the next state s t+1The maximum Q value of all possible actions a′ under α is the learning rate, which controls the size of the update.

[0089] Furthermore, in some embodiments, the second extraction module 40 is configured to extract signal features from the pre-processed physiological signal data and speech signal data based on a deep neural network, and enhance the signal features based on reinforcement learning to obtain text features, including:

[0090] The physiological signal data passes through the preprocessing layer, and the signal features are extracted by the deep neural network, and then the features are enhanced by updating the Q value. The deep neural network includes three hidden layers, namely L1, L2, and L3, and each hidden layer has a corresponding weight matrix W l and the bias vector b l , the activation function is ReLU. The forward propagation process of the deep neural network can be expressed as:

[0091] X l+1 =σ(W l X l +b l )

[0092] Among them, X0 is the input physiological signal data, X l+1 is the output of the lth layer, and σ(.) is the ReLU activation function.

[0093] The backward propagation process of a deep neural network involves the calculation of the loss function and the update of the parameters. Assume that the loss function is Where Y is the true label, is the model's prediction. Commonly used loss functions include mean squared error (MSE) loss and cross extraction (CE) loss. Here, we take mean squared error loss as an example:

[0094]

[0095] Where N is the number of samples.

[0096] For the lth layer, its gradient can be expressed as:

[0097]

[0098] in, is the derivative of the ReLU activation function. For the ReLU activation function, its derivative is:

[0099]

[0100] For the output X of layer l l+1 , whose gradient is:

[0101]

[0102] Substituting the above formula into the calculation of the gradient, we can get:

[0103]

[0104] Finally, the weight matrix W l and the bias vector b l The update rule is:

[0105]

[0106] Here, α is the learning rate.

[0107] Furthermore, in some embodiments, the updating module 50 is configured to update the enhanced image features and text features based on the attention mechanism module to obtain updated image features and updated text features, including:

[0108] The updated image feature data and text data are updated twice through the attention mechanism module. The calculation of the self-attention mechanism involves three matrices: query matrix Q∈R N×D″ , the bond matrix K∈R N×D″ Sum value matrix V∈R N×D′ These matrices can be learned. Self-attention score A∈R N×N It can be calculated by the following formula:

[0109]

[0110] Where T represents transpose, is a normalizing constant.

[0111] Attention weight a i ∈R 1×N It can be obtained by the softmax function:

[0112] a i =softmax(A[:,i])

[0113] The updated feature data X′ can be calculated using the following formula:

[0114]

[0115] Here, ⊙ represents element-wise multiplication.

[0116] Furthermore, in some embodiments, the fusion module 60 is configured to fuse the updated image features and the updated text features to obtain fused feature data, including:

[0117] The image feature data and text data after the second update are fused together through the feature fusion module, which fuses the output data features of different modules. Specifically, it is fused by weighted summation, which fuses information by assigning different weights to the features of different modalities. This disclosure represents the image feature as F X , the text feature is represented as F T , then the fused feature F can be expressed as:

[0118] F=αF X +(1-α)F T

[0119] Among them, α is a weight parameter that can be adjusted as needed.

[0120] Furthermore, in some embodiments, the evaluation module 70 is used to input the fused feature data into the fully connected layer to obtain a motor function evaluation result of the target to be evaluated.

[0121] The fully connected layer outputs the pattern of the patient's movement disorder, providing the patient with an accurate motor function assessment. In motor function assessment, the output F of the feature fusion module can be further processed, for example, by converting it into a probability distribution through a softmax function to predict different patterns of the patient's movement disorder. If our task is a classification task, then the output F′ can be converted through the softmax function to:

[0122] P(y|X)=softmax(F)

[0123] Where y is the patient's movement disorder category, and P(y|X) is the probability of outputting category y given the input data X.

[0124] Figure 3 FIG is a flow chart of a method for evaluating patient motor dysfunction based on multimodal data fusion according to an embodiment of the present disclosure. Figure 3 As shown, the method 200 specifically includes the following steps:

[0125] Step S202 , obtaining multimodal motion performance data of the target to be evaluated, and preprocessing the multimodal motion performance data based on the preprocessing layer; the multimodal motion performance data includes physiological signal data, voice signal data, image data and video data.

[0126] The video data may be input as image data by reading the video data frame by frame.

[0127] Preferably, the signal data and the image data are each preprocessed through a preprocessing layer, including: performing data enhancement, data cleaning and data normalization operations on the multimodal motion performance data.

[0128] Step S204: extracting the human body bounding box and segmentation mask of the pre-processed image data and video data based on the mask residual network.

[0129] Specifically, the Mask Residual Network (Mask_ResNet) detects the bounding box of the human body in the input image, combines the classification and bounding box regression tasks, and uses an additional branch to generate the opposite segmentation mask.

[0130] Step S206 , extracting image features of the human body bounding box and the segmentation mask based on a multi-layer convolutional structure, and enhancing the image features based on reinforcement learning to obtain enhanced image features.

[0131] Step S208: extracting signal features of the pre-processed physiological signal data and speech signal data based on a deep neural network, and enhancing the signal features based on reinforcement learning to obtain text features.

[0132] In step S210, the enhanced image features and text features are updated based on the attention mechanism module to obtain updated image features and updated text features.

[0133] Step S212: Fusing the updated image features and the updated text features to obtain fused feature data.

[0134] In step S214, the fused feature data is input into the fully connected layer to obtain the motor function evaluation result of the target to be evaluated.

[0135] In order to verify the advantages of the multimodal algorithm based on the cross-attention mechanism provided by the embodiment of the present disclosure over the traditional convolutional neural network, an experimental environment as shown in Table 1 was built, and the experimental parameters as shown in Table 2 were set for comparative experiments:

[0136] Table 1 Experimental environment

[0137] project illustrate hardware NVIDIA Tesla V100 GPU x4 operating system Ubuntu 20.04LTS programming language Python 3.8 Deep Learning Framework TensorFlow 2.4 / PyTorch 1.8 Dataset Custom multimodal motion dataset Dataset size 5,000 sports data samples Data preprocessing tools OpenCV, librosa, NumPy

[0138] Table 2 Experimental parameters

[0139]

[0140]

[0141] The comparative experimental results of the multimodal algorithm based on the cross-attention mechanism provided by the embodiment of the present disclosure and the traditional convolutional neural network are shown in Table 3:

[0142] Table 3 Experimental results

[0143]

[0144] As shown in Table 3, the multimodal algorithm based on the cross-attention mechanism provided by the present disclosure outperforms the traditional CNN algorithm in all indicators, especially in terms of accuracy, precision, recall rate and F1 score. Although the training time is slightly increased, the overall performance is significantly improved.

[0145] From the above description, it can be seen that the present disclosure provides a patient movement dysfunction assessment system and method based on multimodal data fusion. It uses multimodal data fusion technology to capture and analyze the patient's movement performance in real time, and identifies the patterns and characteristics of movement disorders through deep learning algorithms. It enhances the complementarity between different modal data through the cross-attention mechanism, improves the effect of feature extraction and fusion, and makes movement function assessment more accurate and efficient, alleviating the technical problem that traditional movement function assessment methods mainly rely on manual observation and subjective judgment, and have great subjectivity and uncertainty.

[0146] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present disclosure.

[0147] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the methods.

[0148] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0149] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present disclosure. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments shown herein, but is intended to be construed in the widest manner consistent with the principles and novel features disclosed herein.

Claims

1. A patient movement dysfunction assessment system based on multimodal data fusion, characterized by: include: an acquisition module, configured to acquire multimodal motion performance data of a target to be evaluated, and preprocess the multimodal motion performance data based on a preprocessing layer; The multimodal motion performance data includes physiological signal data, voice signal data, image data and video data; A first extraction module is used to extract the human body bounding box and segmentation mask of the preprocessed image data and video data based on the mask residual network; An enhancement module, configured to extract image features of the human body bounding box and the segmentation mask based on a multi-layer convolutional structure, and enhance the image features based on reinforcement learning to obtain enhanced image features; A second extraction module is configured to extract signal features from the preprocessed physiological signal data and speech signal data based on a deep neural network, and enhance the signal features based on reinforcement learning to obtain text features; An updating module, configured to update the enhanced image features and the text features based on the attention mechanism module to obtain updated image features and updated text features; A fusion module, configured to fuse the updated image features and the updated text features to obtain fused feature data; The evaluation module is used to input the fused feature data into the fully connected layer to obtain the motor function evaluation result of the target to be evaluated.

2. The system according to claim 1, wherein: in, The preprocessing of the multimodal motion performance data based on the preprocessing layer includes: performing data enhancement, data cleaning and data normalization operations on the multimodal motion performance data.

3. The system according to claim 2, characterized in that: The first extraction module is used to extract the human body bounding box and segmentation mask of the preprocessed image data and video data based on the mask residual network, including: The preprocessed image data and video data are passed through the preprocessing layer. The bounding box of the human body is first detected in the input image through the mask residual network. Then, the classification and bounding box regression tasks are combined and an additional branch is used to generate the segmentation mask of the object.

4. The system according to claim 3, wherein: The overall loss TotalLoss of the mask residual network is: TotalLoss=λ c ×ClassLoss+λ r ×BoxLoss+λ m ×MaskLoss+λ n ×GioULoss; in, Among them, TotalLoss is the overall loss, MaskLoss is the segmentation mask loss, m ij is the ground-truth segmentation mask for the jth pixel of the i-th bounding box, is the predicted segmentation mask probability value; BoxLoss is the bounding box regression loss, t ik is the true value of the kth coordinate (x, y, w, h) of the i-th bounding box, is the predicted value; ClassLoss is the classification loss, y ij is the true label of the jth category of the i-th bounding box, is the predicted probability; λ c ,λ r ,λ m ,λ n is a hyperparameter, b x and b y are the width and height of the predicted bounding box, A pred and A true are the areas of the predicted bounding box and the true bounding box, respectively, and IoU is the ratio of the intersection area of the predicted bounding box and the true bounding box to the union area.

5. The system according to claim 4, characterized in that: The extracting of image features of the human body bounding box and segmentation mask based on a multi-layer convolutional structure includes: Mask_ResNet(x)=ReLU(conv1(x)+skip1)+conv2(ReLU(conv1(x)+skip1)) Among them, Mask_ResNet(x) means inputting the input data x into the multi-layer convolution structure, ReLU is the activation function, conv l represents the convolution operation of the lth layer, x represents the input data, and skip1 represents the skip connection output of the first residual block.

6. The system according to claim 5, characterized in that: in, Enhancing the image features based on reinforcement learning includes: enhancing the image features based on a Q-value function, and the specific process is as follows: Q(s t ,a t )←Q(s t ,a t )+α[R t+1 +γmaxQ(s t+1 ,a′)-Q(s t ,a t )] Where: Q(s t , a t ) is the state s at time step t t Take action a t Q value; R t+1 is the immediate reward obtained at time step t+1; γ is the discount factor used to consider the discount of future returns; maxQ(s t+1 , a′) is in the next state s t+1 The maximum Q value of all possible actions a′ under α is the learning rate, which controls the size of the update.

7. The system according to claim 6, wherein: The deep neural network consists of three hidden layers, namely L1, L2, and L3. Each hidden layer has a corresponding weight matrix W l and the bias vector b l , the activation function is ReLU; The forward propagation process of the deep neural network is expressed as: X l+1 =σ(W l X l +b l ) Among them, X0 is the input physiological signal data, X l+1 is the output of the lth layer, and σ(.) is the ReLU activation function.

8. The system according to claim 7, characterized in that: in, The updated feature data X′ is calculated using the following formula: Among them, the attention weight a j Through the softmax function, we get: j =softmax(A[:,j]), and a j ∈R 1×N ; N represents the total number of image blocks in the image feature data set; x′ j Represents the j-th image feature data x j The corresponding vector; V represents the value matrix of the self-attention mechanism and V∈R N×D ′; D′ is the dimension after flattening; D″ is the dimension of position encoding; ⊙ represents element-wise multiplication; The fused feature data F is expressed as: F=αF X +(1-α)F T Among them, F is the feature data after fusion, F X is the image feature after the update, FT is the text feature after the update, and α is the weight parameter.

9. The system according to claim 8, characterized in that: The evaluation module is used to input the fused feature data into a fully connected layer to obtain a motor function evaluation result of the target to be evaluated, including: The output F of the feature fusion module is converted into a probability distribution through the softmax function to predict different patterns of patient movement disorders. The specific conversion algorithm used is as follows: P(y | X) = softmax(F) Where y is the patient's movement disorder category, and P(y|X) is the probability of outputting category y given the input data X.

10. A method for assessing patient motor dysfunction based on multimodal data fusion, characterized in that: include: Acquiring multimodal motion performance data of a target to be evaluated, and preprocessing the multimodal motion performance data based on a preprocessing layer; The multimodal motion performance data includes physiological signal data, voice signal data, image data and video data; Based on the mask residual network, the human body bounding box and segmentation mask are extracted from the preprocessed image data and video data; Extracting image features of the human body bounding box and the segmentation mask based on a multi-layer convolutional structure, and enhancing the image features based on reinforcement learning to obtain enhanced image features; Extracting signal features from pre-processed physiological signal data and speech signal data based on a deep neural network, and enhancing the signal features based on reinforcement learning to obtain text features; Based on the attention mechanism module, feature updating is performed on the enhanced image features and the text features respectively to obtain updated image features and updated text features; Performing feature fusion on the updated image features and the updated text features to obtain fused feature data; The fused feature data is input into the fully connected layer to obtain the motor function evaluation result of the target to be evaluated.

Images