Motion function rehabilitation prediction method and system based on multi-modal data, terminal and storage medium

Abstract

The invention relates to the technical field of rehabilitation data prediction, and discloses a motion function rehabilitation prediction method and system based on multi-modal data, a terminal and a storage medium, and the method comprises the steps: obtaining an electroencephalogram signal, an electromyographic signal and a functional near infrared spectrum signal of a target user, pre-processing to obtain a target electroencephalogram signal, a target electromyographic signal and a target functional near infrared spectrum signal; respectively carrying out feature extraction to obtain three time-frequency features, and carrying out source positioning processing on the target electroencephalogram signal and the target functional near infrared spectrum signal to obtain brain source intensity; and fusing the three time-frequency features to obtain a target time-frequency feature, inputting the target time-frequency feature and the brain source intensity into a multi-modal data rehabilitation prediction model, and outputting a motor function rehabilitation prediction result. According to the method, the relationship between the three signals of the target user and the motion function of the target user is modeled in the time sequence, so that the long-term evolution of the motion function in the time sequence is quickly evaluated, and the accuracy of a prediction result is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of rehabilitation data prediction, and particularly to a motion function rehabilitation prediction method, system, terminal and storage medium based on multi-modal data. Background Art

[0002] The assessment of human motor function is an important topic in the fields of rehabilitation medicine and sports science. Currently, the main methods for motor function assessment are as follows: Clinical assessment, including observing the patient's motor function, muscle strength, etc. However, it mainly relies on the doctor's experience and observation, may be affected by subjective factors, and it is difficult to quantitatively assess the patient's motor function; Neuroelectrophysiological assessment, although it can provide objective data, may have certain requirements for the comfort and safety of the patient; Motor function assessment scales, although they can provide detailed motor function scores, may be difficult to comprehensively assess all aspects of the patient's motor function; Robot-assisted assessment, which can provide precise measurements of movement trajectories and muscle activities, but may be costly and have high requirements for patients and operators; Among the above assessment methods, clinical assessment, scale assessment, and self-assessment all belong to qualitative assessment and are easily affected by the subjective state of doctors or patients, resulting in inaccurate assessment; while neuroelectrophysiological assessment and robot-assisted assessment are quantitative assessments, but they have the defects of single modality and lack of neural pathway information.

[0003] Therefore, the existing assessment methods mainly rely on single modality or dual modality, lack the application of functional near-infrared spectroscopy signals and their comprehensive use with multiple signals, and lack the capture of long-term correlations between signal time series in most related applications, thus being unable to effectively assess and predict the human motor function of patients, resulting in inaccurate prediction results of human motor function.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0005] The main purpose of the present invention is to provide a motion function rehabilitation prediction method, system, terminal and storage medium based on multi-modal data, aiming to solve the problem in the existing technology that there is a lack of application of functional near-infrared spectroscopy signals and their comprehensive use with multiple signals, and a lack of capture of long-term correlations between signal time series, thus being unable to effectively assess and predict the human motor function of patients, resulting in inaccurate prediction results of human motor function.

[0006] To achieve the above object, the present invention provides a motion function rehabilitation prediction method based on multi-modal data. The motion function rehabilitation prediction method based on multi-modal data includes the following steps: Obtain the electroencephalogram (EEG) signal, electromyogram (EMG) signal, and functional near-infrared spectroscopy (fNIRS) signal of the target user, and preprocess the EEG signal, EMG signal, and fNIRS signal to obtain a target EEG signal, a target EMG signal, and a target fNIRS signal; Extract features from the target EEG signal, the target EMG signal, and the target fNIRS signal respectively to obtain first time-frequency features, second time-frequency features, and third time-frequency features, and perform source localization processing on the target EEG signal and the target fNIRS signal to obtain brain source intensity; Fuse the first time-frequency features, the second time-frequency features, and the third time-frequency features to obtain target time-frequency features, and input the target time-frequency features and the brain source intensity into a trained multi-modal data rehabilitation prediction model to output the motion function rehabilitation prediction result of the target user.

[0007] Optionally, in the motion function rehabilitation prediction method based on multi-modal data, where the step of obtaining the EEG signal, EMG signal, and fNIRS signal of the target user, and preprocessing the EEG signal, EMG signal, and fNIRS signal to obtain a target EEG signal, a target EMG signal, and a target fNIRS signal specifically includes: Set an experimental paradigm according to a clinical assessment scale, and set a first sampling frequency, a second sampling frequency, and a third sampling frequency; Collect the EEG signal, EMG signal, and fNIRS signal of the target user within a preset period according to the experimental paradigm, the first sampling frequency, the second sampling frequency, and the third sampling frequency; Perform filtering, artifact removal, and data segmentation on the EEG signal and the EMG signal to obtain a target EEG signal and a target EMG signal, and perform filtering and conversion processing on the fNIRS signal to obtain a target fNIRS signal.

[0008] Optionally, in the motion function rehabilitation prediction method based on multi-modal data, where the step of extracting features from the target EEG signal, the target EMG signal, and the target fNIRS signal respectively to obtain first time-frequency features, second time-frequency features, and third time-frequency features specifically includes: Perform signal subtraction on the target EMG signal to obtain an amplitude difference value, select channels according to the amplitude difference value to obtain a first channel, and extract features from the target EMG signal according to the first channel to obtain second time-frequency features; Select channels from the target EEG signal and the target fNIRS signal respectively through a linear model to obtain a second channel and a third channel; Feature extraction is performed on the target EEG signal according to the second channel to obtain first time-frequency features, and feature extraction is performed on the target functional near-infrared spectroscopy signal according to the third channel to obtain third time-frequency features.

[0009] Optionally, in the motion function rehabilitation prediction method based on multimodal data, the expression of the linear model is: ; Where is the matrix of measurement data, is the number of data points, M is the number of channels, is the design matrix, L is the number of conditions, is the regression coefficient matrix to be estimated, is the residual error matrix.

[0010] Optionally, in the motion function rehabilitation prediction method based on multimodal data, the source localization process for the target EEG signal and the target functional near-infrared spectroscopy signal to obtain brain source intensity specifically includes: Performing current estimation on the target EEG signal to obtain a likelihood function, and determining the mean parameter of the inverse variance of the brain current source according to the target functional near-infrared spectroscopy signal; Obtaining the inverse variance parameter, setting the prior probability distribution of the inverse variance parameter according to the mean parameter of the inverse variance of the brain current source, and determining the prior probability distribution of the source current according to the prior probability distribution of the inverse variance parameter; Obtaining a posterior distribution according to the prior probability distribution of the source current and the likelihood function, and performing variational inference processing on the posterior distribution to obtain a variational distribution; Maximizing the lower bound of the variational distribution to obtain a target lower bound, and performing data localization on the variational distribution according to the target lower bound to obtain brain source intensity.

[0011] Optionally, in the motion function rehabilitation prediction method based on multimodal data, the training process of the multimodal data rehabilitation prediction model specifically includes: Obtaining a training data set, where the training data set includes multiple groups of training samples, and each group of training samples includes the time-frequency features of historical EEG signals, the time-frequency features of historical electromyography signals, the time-frequency features of historical functional near-infrared spectroscopy signals, and historical brain source intensity; Creating a multimodal data rehabilitation training model, inputting a group of training samples into the multimodal data rehabilitation training model, and performing embedding processing on the training samples to obtain target training samples; Perform a linear transformation on the target training samples to obtain query vectors, key vectors, and value vectors. Calculate the dot products of the query vectors and the key vectors to obtain first and second dot products, and normalize the first and second dot products to obtain a probability distribution; Perform a weighted calculation on the probability distribution according to the value vectors to obtain a target weight sum. Obtain a motion function rehabilitation evaluation prediction result according to the target weight sum, and calculate a loss according to the motion function rehabilitation evaluation prediction result and the true label sequence corresponding to the training samples to obtain a loss value. Modify the internal parameters of the multi-modal data rehabilitation training model according to the loss value; Input the next set of training samples into the multi-modal data rehabilitation training model until the training condition of the multi-modal data rehabilitation training model meets the preset condition to obtain the multi-modal data rehabilitation prediction model.

[0012] Optionally, in the motion function rehabilitation prediction method based on multi-modal data, the calculation formula of the loss value is: ; Wherein, is the loss value, is the total number of cycles of the input signal, is the number of cycles, is the motion function rehabilitation evaluation prediction result, is the true label sequence corresponding to the training samples.

[0013] Optionally, in the motion function rehabilitation prediction method based on multi-modal data, the motion function rehabilitation prediction system based on multi-modal data includes: A signal acquisition module, configured to acquire electroencephalogram signals, electromyogram signals, and functional near-infrared spectroscopy signals of a target user, and preprocess the electroencephalogram signals, the electromyogram signals, and the functional near-infrared spectroscopy signals to obtain target electroencephalogram signals, target electromyogram signals, and target functional near-infrared spectroscopy signals; A feature extraction module, configured to respectively extract features from the target electroencephalogram signals, the target electromyogram signals, and the target functional near-infrared spectroscopy signals to obtain first time-frequency features, second time-frequency features, and third time-frequency features, and perform source localization processing on the target electroencephalogram signals and the target functional near-infrared spectroscopy signals to obtain brain source intensities; A result prediction module, configured to fuse the first time-frequency features, the second time-frequency features, and the third time-frequency features to obtain target time-frequency features, and input the target time-frequency features and the brain source intensities into a trained multi-modal data rehabilitation prediction model to output a motion function rehabilitation prediction result of the target user.

[0014] In addition, to achieve the above object, the present invention further provides a terminal, wherein the terminal includes: a memory, a processor, and a program stored on the memory and executable on the processor, and when the program is executed by the processor, the steps of the above-mentioned motion function rehabilitation prediction method based on multimodal data are implemented.

[0015] In addition, to achieve the above object, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a motion function rehabilitation prediction program based on multimodal data, and when the motion function rehabilitation prediction program based on multimodal data is executed by a processor, the steps of the above-mentioned motion function rehabilitation prediction method based on multimodal data are implemented.

[0016] In the present invention, an electroencephalogram signal, an electromyogram signal, and a functional near-infrared spectroscopy signal of a target user are acquired, and the electroencephalogram signal, the electromyogram signal, and the functional near-infrared spectroscopy signal are preprocessed to obtain a target electroencephalogram signal, a target electromyogram signal, and a target functional near-infrared spectroscopy signal; feature extraction is respectively performed on the target electroencephalogram signal, the target electromyogram signal, and the target functional near-infrared spectroscopy signal to obtain a first time-frequency feature, a second time-frequency feature, and a third time-frequency feature, and source localization processing is performed on the target electroencephalogram signal and the target functional near-infrared spectroscopy signal to obtain brain source intensity; the first time-frequency feature, the second time-frequency feature, and the third time-frequency feature are feature-fused to obtain a target time-frequency feature, and the target time-frequency feature and the brain source intensity are input into a trained multimodal data rehabilitation prediction model to output a motion function rehabilitation prediction result of the target user. The present invention improves the accuracy of the prediction result by mining multiple information to construct a multimodal data rehabilitation prediction model for the motion function of people with movement disorders and modeling the relationship between the electroencephalogram signal, electromyogram signal, and functional near-infrared spectroscopy signal of the target user and their motion function in time series to quickly evaluate the long-term evolution of the motion function in time series. Description of the Drawings

[0017] Figure 1 is a flowchart of a preferred embodiment of the motion function rehabilitation prediction method based on multimodal data in the present invention; Figure 2 is a schematic diagram of the overall process of the motion function rehabilitation prediction method based on multimodal data in the present invention; Figure 3 is a schematic diagram of the network structure of a multimodal data rehabilitation training model of a preferred embodiment of the present invention; Figure 4 is a structural diagram of a preferred embodiment of the motion function rehabilitation prediction system based on multimodal data in the present invention; Figure 5It is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed implementation manners

[0018] To make the objectives, technical solutions and advantages of the present invention clearer and more definite, the present invention will be further described in detail below with reference to the accompanying drawings and by way of examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0019] It should be noted that if there are directional indications (such as up, down, left, right, front, back,...) involved in the embodiments of the present invention, the directional indications are only used to explain the relative positional relationship and movement conditions between components in a specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications will also change accordingly.

[0020] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between various embodiments may be combined with each other, but it must be based on the fact that those of ordinary skill in the art can implement them. When the combination of technical solutions results in contradictions or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope required by the present invention.

[0021] The method for predicting the motor function rehabilitation based on multimodal data according to a preferred embodiment of the present invention, as Figure 1 shown, the method for predicting the motor function rehabilitation based on multimodal data includes the following steps: Step S10: Obtain the electroencephalogram signal, electromyogram signal and functional near-infrared spectroscopy signal of a target user, and preprocess the electroencephalogram signal, the electromyogram signal and the functional near-infrared spectroscopy signal to obtain a target electroencephalogram signal, a target electromyogram signal and a target functional near-infrared spectroscopy signal.

[0022] The step S10 includes: Step S11: Set an experimental paradigm according to a clinical assessment scale, and set a first sampling frequency, a second sampling frequency and a third sampling frequency; Step S12: Collect the electroencephalogram signal, the electromyogram signal and the functional near-infrared spectroscopy signal of the target user within a preset period according to the experimental paradigm, the first sampling frequency, the second sampling frequency and the third sampling frequency; Step S13: Filter the EEG signals and EMG signals, remove artifacts, and segment the data to obtain target EEG signals and target EMG signals. Filter and transform the functional near-infrared spectroscopy signals to obtain target functional near-infrared spectroscopy signals.

[0023] Specifically, existing methods mainly focus on single-modal or dual-modal, lacking the application of functional near-infrared spectroscopy signals and their comprehensive use with multiple signals. Most of them only use common amplitude and frequency features of signals, lacking in-depth exploration and use of signal time and space information. Moreover, neural networks lack the capture of long-term correlations between signal time series in most related applications. In contrast, the present invention comprehensively uses synchronously collected EEG signals, EMG signals, and functional near-infrared spectroscopy signals for comprehensive evaluation and prediction. It extracts time-frequency features from signals after key channel selection to help the model learn quickly, performs comprehensive source localization on all-channel brain signals to complement information loss caused by feature extraction, and uses neural networks to autonomously learn the two types of signal features, thereby realizing the rehabilitation evaluation and prediction of the motor function of disabled people. In the embodiments of the present invention, at specific periodic intervals (e.g., every week), EMG signals, EEG signals, and functional near-infrared spectroscopy signals of target users (e.g., people with motor disabilities) are synchronously collected under a predefined experimental paradigm, and the motor function score results obtained by physicians based on scales during the evaluation of target users are synchronously recorded, i.e., the motor function score. Then, data preprocessing, channel selection, feature extraction, and source localization are performed on the EMG signals, EEG signals, and functional near-infrared spectroscopy signals. Subsequently, a model is established based on the attention mechanism network, and motor evaluation is carried out according to the fused feature array. The specific process is as Figure 2 shown.

[0024] First, set up an experimental paradigm according to a clinical assessment scale. Among them, the experimental paradigm includes upper limb movement paradigms, lower limb movement paradigms, the collection sites and durations of electromyogram (EMG) signals, the collection sites and durations of electroencephalogram (EEG) signals, the collection sites and durations of functional near-infrared spectroscopy (fNIRS) signals, the clinical assessment cycle, and the experimental collection cycle. Conduct the experimental paradigm at regular intervals (for example, every other week), record the scoring results obtained by physicians' evaluation of the target user according to the scale, and set the first sampling frequency (i.e., the sampling frequency of EEG signals, commonly used sampling frequencies are 250 Hz, 500 Hz, or 1000 Hz), the second sampling frequency (i.e., the sampling frequency of EMG signals, commonly used sampling frequencies are 500 Hz or 1000 Hz), and the third sampling frequency (i.e., the sampling frequency of fNIRS signals, commonly used sampling frequency is 10 Hz). Synchronously collect the EEG signals, EMG signals, and fNIRS signals of the target user according to the first sampling frequency, the second sampling frequency, and the third sampling frequency. However, during the signal collection process, many interference signals will be encountered, such as electrooculogram (EOG) signals, heartbeat artifacts, and interference from power frequency sources. These interferences will reduce the signal-to-noise ratio, thus having a negative impact on the accuracy of subsequent feature extraction and prediction evaluation. Therefore, it is necessary to preprocess the synchronously collected EEG signals, EMG signals, and fNIRS signals to obtain target EEG signals, target EMG signals, and target fNIRS signals. Specifically, for the EEG signals and EMG signals, the main steps are filtering, artifact removal, and data segmentation. For the fNIRS signals, first filter to eliminate physiological noise, and then use the improved Beer–Lambert law to convert the changes in the original optical density signals into the changes in oxyhemoglobin concentration at each time point and deoxyhemoglobin concentration changes , and the corresponding expressions are: ; where and are the extinction coefficients of oxyhemoglobin at wavelengths and respectively, and are the extinction coefficients of deoxyhemoglobin at wavelengths and respectively, and are the optical density changes at wavelengths and respectively, and the calculation formula for the optical density change is: ; where is the optical density change at wavelength ​ as the reference signal at wavelength , and as the detection signal at wavelength .

[0025] Step S20: Feature extraction is performed on the target electroencephalogram signal, the target electromyogram signal, and the target functional near-infrared spectroscopy signal respectively to obtain the first time-frequency feature, the second time-frequency feature, and the third time-frequency feature, and source localization processing is performed on the target electroencephalogram signal and the target functional near-infrared spectroscopy signal to obtain the brain source intensity.

[0026] Specifically, after obtaining the target electroencephalogram signal, the target electromyogram signal, and the target functional near-infrared spectroscopy signal, channel selection needs to be performed according to different rules to eliminate redundant or low-quality information to ensure the accuracy of the evaluation system. The specific process is as follows: for the target electromyogram signal, the minimum value of the target electromyogram signal is subtracted from the median of each electrode signal of the target user to obtain the amplitude difference; the average value of the amplitude differences of all muscle electrodes of the target user is taken, and the smallest average value is used as the threshold for channel selection to obtain the first channel. For the target electroencephalogram signal and the target functional near-infrared spectroscopy signal, a linear model is used for channel selection to obtain the second channel and the third channel. Among them, the expression of the linear model is: ; where is the matrix of measurement data, is the number of data points, M is the number of channels, is the design matrix, L is the number of conditions, including tasks and any terms regarded as related to the data variance source, is the matrix of regression coefficients to be estimated, is the residual error matrix.

[0027] For the signal, a one-sample test is used to test and to identify values, so as to represent the channels with significant contrast between movement execution tasks. The calculation formula of this value is: ; where c is the contrast vector, which determines the contrast between specific conditions, is the total number of cycles of the input signal. The , select a group from the channels with significant contrasts between the motor execution tasks as candidate channels; for the left and right brain regions, select the functional near-infrared spectroscopy channels that produce higher values, and their adjacent electroencephalogram channels (i.e., the second channels).

[0028] After that, divide the processed data according to a fixed time window length (e.g., 10 ms, 50 ms, etc.) to extract features, obtaining the first time-frequency feature, the second time-frequency feature, and the third time-frequency feature. For the first time-frequency feature and the second time-frequency feature, extract the root mean square RMS and the average absolute value MAV of the target electromyogram signal of the first channel and the target electroencephalogram signal of the second channel within the time window length. The corresponding extraction formulas are: ; ; where N is the total number of data points within the fixed time window, is the value of the signal at time point . Subsequently, perform Fourier transform on the target electroencephalogram signal and the target electromyogram signal, calculate the power spectral density , and then calculate the median frequency MF and the average power frequency MPF of the feature values. The corresponding calculation formulas are: ; ; where is the frequency value, is the highest frequency of the signal. Similarly, for the oxyhemoglobin and deoxyhemoglobin concentration change signals converted from functional near-infrared spectroscopy, extract the mean SM , slope SLP , kurtosis KRT , skewness SKW and other features. The corresponding extraction formulas are: ; ; ; ; where is the value of the signal at time point N , is the mean value of the signal in this time window, is the standard deviation of the signal in this time window.

[0029] Subsequently, comprehensive source localization is performed on the full-channel brain signals to complement the information loss caused by feature extraction and obtain more comprehensive information. Specifically, multi-modal source localization under the variational Bayesian framework is performed on the target electroencephalogram (EEG) signals and the target functional near-infrared spectroscopy (fNIRS) signals to achieve in-depth signal mining. Assume that is the source activity current density (i.e., source strength) on the cerebral cortex. For the source localization of the target EEG signals, a current source distribution model is used, and the corresponding expression is: ; where, is the scalp EEG signal, G is the lead field matrix, that is, the spatial relationship between the scalp EEG signal and the source current, is the noise term, which is set to follow a Gaussian distribution with a mean of zero and a spherical covariance. Hierarchical Bayesian is used to estimate the current of the target EEG signal to obtain the likelihood function , which is expressed as: ; where, is the reciprocal of the EEG noise variance. For the prior probability distribution of the source current, a normal prior is first set, which is expressed as: ; where, is the transpose of, is the inverse variance parameter of the unknown brain current source, is its diagonal matrix, and the prior probability distribution of the inverse variance parameter is: ; where, is a Gamma distribution with a mean of and a degree of freedom of . is set to 10. For the unknown parameter (i.e., the inverse variance mean parameter of the brain current source), the target fNIRS data is used to determine it, and the corresponding expression is: ; where, is the baseline of the previous current variance, which is obtained by using the minimum norm estimation from the baseline interval of the EEG data, is the normalized activity data of the target fNIRS data at the th moment, is the variance amplification factor, which is usually set to 100.

[0030] After determining the prior distribution, the best result is found by maximizing the following posterior distribution The expression is: ; where is the likelihood function, that is, the probability of the EEG data appearing when the source strength is given, is the prior distribution of the source current, is the marginal probability of the EEG data . Since is difficult to calculate directly, the method of variational inference is introduced here to approximate its posterior distribution as a parameterized variational distribution , and the true posterior is approximated by maximizing the lower bound of the variational distribution , and the corresponding formula is: ; where is the logarithmic expectation of the joint probability of the EEG data E and the source strength , is the entropy of the variational distribution, which is used to prevent overfitting. Subsequently, the lower bound of the variational distribution is optimized by the logarithmic expectation maximization method. Specifically, fixing other parameters, is maximized to obtain the optimal variational distribution , and the calculation formula is: ; After that, fixing the optimal variational distribution, the estimation of is optimized to obtain the brain source strength , and the corresponding expression is: .

[0031] Step S30: Perform feature fusion on the first time-frequency feature, the second time-frequency feature, and the third time-frequency feature to obtain a target time-frequency feature, and input the target time-frequency feature and the brain source strength into the trained multi-modal data rehabilitation prediction model to output the motion function rehabilitation prediction result of the target user.

[0032] Specifically, in the embodiments of the present invention, the first time-frequency feature, the second time-frequency feature, and the third time-frequency feature are subjected to feature fusion to obtain a target time-frequency feature. Then, the target time-frequency feature and the brain source intensity are input into a trained multi-modal data rehabilitation prediction model. Before inputting into the multi-modal data rehabilitation prediction model, it is necessary to train a multi-modal data rehabilitation training model (i.e., the Transformer Encoder network model) to obtain the multi-modal data rehabilitation prediction model. The network structure of the multi-modal data rehabilitation training model is as Figure 3 shown. This multi-modal data rehabilitation training model uses a parallel network architecture: First, the multi-modal data rehabilitation training model performs an embedding operation on the original feature input data for each time step, using a linear layer to map it to a higher-dimensional space, which is more suitable for the multi-modal data rehabilitation training model to learn. Subsequently, the multi-modal data rehabilitation training model adds a position encoding corresponding to the input time step length to each data to represent the time order, so that the multi-modal data rehabilitation training model can learn the temporal features. After the existence of the position encoding, the feature sequences of each time step can be input into the same network model in parallel to participate in the operation together. This can not only improve the training speed of the algorithm, but also make up for the deficiency of traditional network models in forgetting early information, capture long-term dependencies, and enhance the expression ability of the model. The specific training process is as follows: Obtain a training data set, where the training data set includes multiple groups of training samples, and each group of training samples includes the time-frequency features of historical electroencephalogram signals, the time-frequency features of historical electromyogram signals, the time-frequency features of historical functional near-infrared spectroscopy signals, and the historical brain source intensity; input a group of training samples into the multi-modal data rehabilitation training model, and perform embedding processing on the training samples through an encoder to obtain target training samples.

[0033] For the encoder part, it is composed of n identical layers stacked together, usually 2-6 layers. When calculating the self-attention of the encoder, the data all come from the output of the previous encoder layer. Each encoder layer has two sub-layers, namely multi-head self-attention aggregation and position-based feed-forward network. Each sub-layer uses a residual connection, enabling the input to propagate forward faster through the cross-layer data line. This also requires that for any input x at any position in the sequence, the sub-layer output sublayer(x) is required to facilitate the calculation of the residual connection. Therefore, for each position corresponding to the input sequence, the encoder will output a d-dimensional representation vector. After the addition calculation of the residual connection, the model immediately applies layer normalization to achieve magnitude unification based on the feature dimension and improve the accuracy. Self-attention is the core mechanism of Transformer, aiming to capture dependencies between each time step of the input. Each time step feature of the input is a vector , the main steps of self-attention include: for the input at each time step, the model will obtain three vectors respectively through linear transformation: the query vector , the key vector and the value vector , where the three weight matrices of the linear transformation are trainable parameters, that is, performing linear transformation on the target training samples to obtain the query vector, the key vector and the value vector; subsequently, for each time step, calculate and 's dot product to obtain the first dot product and the second dot product, and convert the scores into a probability distribution through normalization, which reflects the relationship strength between two time steps, that is, performing normalization processing on the first dot product and the second dot product to obtain a probability distribution. Finally, calculate the weighted sum obtained by each weighting the probability distribution as the output of self-attention, which represents that the time step combines the context information of all other time steps, while the sub-layer multi-head attention uses multiple attention heads to simultaneously focus on different relationships and patterns. Each attention head learns different query mappings, key mappings and value mappings respectively, and calculates multiple different weighted sum values; and splices the outputs of all attention heads and performs linear transformation to obtain the final output, that is, the motion function rehabilitation evaluation prediction result, represented by , , where the number of attention heads can be set to 3 - 8.

[0034] The self-attention layer captures the relationships between different time steps, but the features of each time step still need to be further non-linearly transformed through a feed-forward network to improve the model's expressive ability. And the position-based feed-forward network uses the same multi-layer perceptron when transforming the representations of all positions in the sequence, which is why the feed-forward network is called position-based. Then, according to the motion function rehabilitation evaluation prediction result and the true label sequence corresponding to the training sample (that is , ), calculate the loss to obtain a loss value, where the calculation formula of the loss value is: ; is the loss value, is the total number of cycles of the input signal; then, correct the internal parameters of the multi-modal data rehabilitation training model according to the loss value; input the next set of training samples into the multi-modal data rehabilitation training model until the training situation of the multi-modal data rehabilitation training model meets the preset conditions to obtain the multi-modal data rehabilitation prediction model.

[0035] After obtaining the trained multi-modal data rehabilitation prediction model, input the target time-frequency feature and the brain source intensity into the trained multi-modal data rehabilitation prediction model to output the prediction result of the target user's motor function rehabilitation, so as to effectively evaluate and predict the evolution of the target user's human motor function rehabilitation over time, and improve the accuracy of the target user's motor function prediction. The present invention mines multiple information to construct a multi-modal data rehabilitation prediction model for the motor function of people with movement disorders, and models the relationship between the electroencephalogram signal, electromyogram signal and functional near-infrared spectroscopy signal of the target user and their motor function over time, so as to quickly evaluate the long-term evolution of the motor function over time, thereby improving the accuracy of the prediction result.

[0036] Further, as Figure 4 shown, based on the above-mentioned method for predicting motor function rehabilitation based on multi-modal data, the present invention also correspondingly provides a system for predicting motor function rehabilitation based on multi-modal data. The system for predicting motor function rehabilitation based on multi-modal data includes: A signal acquisition module 51, configured to acquire the electroencephalogram signal, electromyogram signal and functional near-infrared spectroscopy signal of the target user, and preprocess the electroencephalogram signal, the electromyogram signal and the functional near-infrared spectroscopy signal to obtain a target electroencephalogram signal, a target electromyogram signal and a target functional near-infrared spectroscopy signal; A feature extraction module 52, configured to respectively extract features from the target electroencephalogram signal, the target electromyogram signal and the target functional near-infrared spectroscopy signal to obtain a first time-frequency feature, a second time-frequency feature and a third time-frequency feature, and perform source localization processing on the target electroencephalogram signal and the target functional near-infrared spectroscopy signal to obtain a brain source intensity; A result prediction module 53, configured to fuse the first time-frequency feature, the second time-frequency feature and the third time-frequency feature to obtain a target time-frequency feature, and input the target time-frequency feature and the brain source intensity into the trained multi-modal data rehabilitation prediction model to output the prediction result of the target user's motor function rehabilitation.

[0037] Further, as Figure 5 shown, based on the above-mentioned method for predicting motor function rehabilitation based on multi-modal data, the present invention also correspondingly provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 5 Only some components of the terminal are shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented.

[0038] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as the hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc., equipped on the terminal. Further, the memory 20 may also include both the internal storage unit and the external storage device of the terminal. The memory 20 is used to store application software installed on the terminal and various types of data, such as the program code of the installed terminal. The memory 20 may also be used to temporarily store data that has been output or will be output. In one embodiment, a motion function rehabilitation prediction program 40 based on multimodal data is stored on the memory 20, and the motion function rehabilitation prediction program 40 based on multimodal data can be executed by the processor 10, so as to implement the motion function rehabilitation prediction method based on multimodal data in this application.

[0039] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chips, and is used to run the program code stored in the memory 20 or process data, such as executing the motion function rehabilitation prediction method based on multimodal data, etc.

[0040] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch liquid crystal display, and an OLED (Organic Light-Emitting Diode) toucher, etc. The display 30 is used to display information on the terminal and to display a visual user interface.

[0041] In one embodiment, when the processor 10 executes the program 40 for motion function rehabilitation prediction based on multimodal data in the memory 20, the following steps are implemented: Obtain the electroencephalogram signal, electromyogram signal, and functional near-infrared spectroscopy signal of the target user, and preprocess the electroencephalogram signal, the electromyogram signal, and the functional near-infrared spectroscopy signal to obtain a target electroencephalogram signal, a target electromyogram signal, and a target functional near-infrared spectroscopy signal; Extract features from the target electroencephalogram signal, the target electromyogram signal, and the target functional near-infrared spectroscopy signal respectively to obtain first time-frequency features, second time-frequency features, and third time-frequency features, and perform source localization processing on the target electroencephalogram signal and the target functional near-infrared spectroscopy signal to obtain brain source intensity; Fuse the first time-frequency feature, the second time-frequency feature, and the third time-frequency feature to obtain a target time-frequency feature, and input the target time-frequency feature and the brain source intensity into a trained multi-modal data rehabilitation prediction model to output the motion function rehabilitation prediction result of the target user.

[0042] Among them, the steps of acquiring the electroencephalogram (EEG) signal, electromyogram (EMG) signal, and functional near-infrared spectroscopy (fNIRS) signal of the target user, and preprocessing the EEG signal, the EMG signal, and the fNIRS signal to obtain a target EEG signal, a target EMG signal, and a target fNIRS signal specifically include: Set an experimental paradigm according to a clinical assessment scale, and set a first sampling frequency, a second sampling frequency, and a third sampling frequency; Collect the EEG signal, EMG signal, and fNIRS signal of the target user within a preset period according to the experimental paradigm, the first sampling frequency, the second sampling frequency, and the third sampling frequency; Perform filtering, artifact removal, and data segmentation on the EEG signal and the EMG signal to obtain a target EEG signal and a target EMG signal, and perform filtering and conversion processing on the fNIRS signal to obtain a target fNIRS signal.

[0043] Among them, the steps of respectively extracting features from the target EEG signal, the target EMG signal, and the target fNIRS signal to obtain a first time-frequency feature, a second time-frequency feature, and a third time-frequency feature specifically include: Perform signal subtraction on the target EMG signal to obtain an amplitude difference value, select channels according to the amplitude difference value to obtain a first channel, and extract features from the target EMG signal according to the first channel to obtain a second time-frequency feature; Select channels from the target EEG signal and the target fNIRS signal respectively through a linear model to obtain a second channel and a third channel; Extract features from the target EEG signal according to the second channel to obtain a first time-frequency feature, and extract features from the target fNIRS signal according to the third channel to obtain a third time-frequency feature.

[0044] Among them, the expression of the linear model is: ; Among them, is the matrix of measurement data, is the number of data points, M is the number of channels, is the design matrix,L is the number of conditions, is the regression coefficient matrix to be estimated, is the residual error matrix.

[0045] Among them, the source localization processing of the target electroencephalogram signal and the target functional near-infrared spectroscopy signal to obtain the brain source intensity specifically includes: Estimate the current of the target electroencephalogram signal to obtain a likelihood function, and determine the mean parameter of the inverse variance of the brain current source according to the target functional near-infrared spectroscopy signal; Obtain the inverse variance parameter, set the prior probability distribution of the inverse variance parameter according to the mean parameter of the inverse variance of the brain current source, and determine the prior probability distribution of the source current according to the prior probability distribution of the inverse variance parameter; Obtain the posterior distribution according to the prior probability distribution of the source current and the likelihood function, and perform variational inference processing on the posterior distribution to obtain a variational distribution; Maximize the lower bound of the variational distribution to obtain a target lower bound, and perform data localization on the variational distribution according to the target lower bound to obtain the brain source intensity.

[0046] Among them, the training process of the multimodal data rehabilitation prediction model specifically includes: Obtain a training data set, where the training data set includes multiple groups of training samples, and each group of training samples includes the time-frequency characteristics of historical electroencephalogram signals, the time-frequency characteristics of historical electromyogram signals, the time-frequency characteristics of historical functional near-infrared spectroscopy signals, and historical brain source intensities; Create a multimodal data rehabilitation training model, input a group of training samples into the multimodal data rehabilitation training model, and perform embedding processing on the training samples to obtain target training samples; Perform a linear transformation on the target training samples to obtain query vectors, key vectors, and value vectors, calculate the dot products of the query vectors and the key vectors to obtain a first dot product and a second dot product, and perform normalization processing on the first dot product and the second dot product to obtain a probability distribution; Perform weighted calculation on the probability distribution according to the value vectors to obtain a target weight sum, obtain a motion function rehabilitation evaluation prediction result according to the target weight sum, calculate a loss value according to the motion function rehabilitation evaluation prediction result and the true label sequence corresponding to the training sample, and correct the internal parameters of the multimodal data rehabilitation training model according to the loss value; Input the next group of training samples into the multimodal data rehabilitation training model until the training situation of the multimodal data rehabilitation training model meets the preset conditions to obtain the multimodal data rehabilitation prediction model.

[0047] Among them, the calculation formula of the loss value is as follows: ; Among them, is the loss value, is the total number of cycles of the input signal, is the number of cycles, is the prediction result of the motion function rehabilitation assessment, is the true label sequence corresponding to the training sample.

[0048] The present invention also provides a computer-readable storage medium. Among them, the computer-readable storage medium stores a motion function rehabilitation prediction program based on multimodal data. When the motion function rehabilitation prediction program based on multimodal data is executed by a processor, the steps of the motion function rehabilitation prediction method based on multimodal data as described above are implemented.

[0049] It should be noted that in this article, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, article or device including the element.

[0050] Of course, those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware (such as a processor, a controller, etc.) through a computer program. The program can be stored in a computer-readable computer-readable storage medium, and the program can include the processes of the above method embodiments when executed. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0051] It should be understood that the application of the present invention is not limited to the above examples. For those of ordinary skill in the art, improvements or transformations can be made according to the above description, and all such improvements and transformations should fall within the protection scope of the appended claims of the present invention.

Claims

1. A motion function rehabilitation prediction method based on multi-modal data, characterized in that, The motion function rehabilitation prediction method based on multimodal data includes: Obtain the electroencephalogram (EEG) signal, electromyogram (EMG) signal, and functional near-infrared spectroscopy (fNIRS) signal of the target user, and preprocess the EEG signal, EMG signal, and fNIRS signal to obtain a target EEG signal, a target EMG signal, and a target fNIRS signal; Respectively extract features from the target EEG signal, the target EMG signal, and the target fNIRS signal to obtain first time-frequency features, second time-frequency features, and third time-frequency features, and perform source localization processing on the target EEG signal and the target fNIRS signal to obtain brain source intensity; Fuse the first time-frequency features, the second time-frequency features, and the third time-frequency features to obtain target time-frequency features, and input the target time-frequency features and the brain source intensity into a trained multimodal data rehabilitation prediction model to output the motion function rehabilitation prediction result of the target user.

2. The method for predicting motor function rehabilitation based on multimodal data according to claim 1, wherein The obtaining of the EEG signal, EMG signal, and fNIRS signal of the target user, and the preprocessing of the EEG signal, EMG signal, and fNIRS signal to obtain a target EEG signal, a target EMG signal, and a target fNIRS signal specifically includes: Set an experimental paradigm according to a clinical assessment scale, and set a first sampling frequency, a second sampling frequency, and a third sampling frequency; Collect the EEG signal, EMG signal, and fNIRS signal of the target user within a preset period according to the experimental paradigm, the first sampling frequency, the second sampling frequency, and the third sampling frequency; Perform filtering processing, artifact removal, and data segmentation on the EEG signal and the EMG signal to obtain a target EEG signal and a target EMG signal, and perform filtering processing and conversion processing on the fNIRS signal to obtain a target fNIRS signal.

3. The method for predicting motor function rehabilitation based on multimodal data according to claim 1, wherein The respectively extracting features from the target EEG signal, the target EMG signal, and the target fNIRS signal to obtain first time-frequency features, second time-frequency features, and third time-frequency features specifically includes: Perform signal difference on the target EMG signal to obtain an amplitude difference value, select channels according to the amplitude difference value to obtain a first channel, and extract features from the target EMG signal according to the first channel to obtain second time-frequency features; Respectively select channels from the target EEG signal and the target fNIRS signal through a linear model to obtain a second channel and a third channel; Extract features from the target EEG signal according to the second channel to obtain first time-frequency features, and extract features from the target fNIRS signal according to the third channel to obtain third time-frequency features.

4. The method for predicting motor function rehabilitation based on multimodal data according to claim 3, wherein The expression of the linear model is: ； wherein, is the matrix of measurement data, is the number of data points, M is the number of channels, is the design matrix, L is the number of conditions, is the regression coefficient matrix to be estimated, is the residual error matrix.

5. The method for predicting the rehabilitation of motor function based on multimodal data according to claim 1, wherein, The performing of source localization processing on the target EEG signal and the target fNIRS signal to obtain brain source intensity specifically includes: Perform current estimation on the target EEG signal to obtain a likelihood function, and determine the inverse variance mean parameter of the brain current source according to the target fNIRS signal; Obtain the inverse variance parameter, set the prior probability distribution of the inverse variance parameter according to the inverse variance mean parameter of the brain current source, and determine the prior probability distribution of the source current according to the prior probability distribution of the inverse variance parameter; Obtain the posterior distribution according to the prior probability distribution of the source current and the likelihood function, and perform variational inference processing on the posterior distribution to obtain the variational distribution; Maximize the lower bound of the variational distribution to obtain the target lower bound, and perform data localization on the variational distribution according to the target lower bound to obtain the brain source intensity.

6. The method for predicting motor function rehabilitation based on multimodal data according to claim 1, wherein The training process of the multi-modal data rehabilitation prediction model specifically includes: Obtain a training data set, where the training data set includes multiple groups of training samples, and each group of training samples includes the time-frequency characteristics of historical electroencephalogram signals, the time-frequency characteristics of historical electromyogram signals, the time-frequency characteristics of historical functional near-infrared spectroscopy signals, and historical brain source intensity; Create a multi-modal data rehabilitation training model, input a group of training samples into the multi-modal data rehabilitation training model, and perform embedding processing on the training samples to obtain target training samples; Perform a linear transformation on the target training samples to obtain query vectors, key vectors, and value vectors, calculate the dot product of the query vectors and the key vectors to obtain the first dot product and the second dot product, and perform normalization processing on the first dot product and the second dot product to obtain a probability distribution; Perform a weighted calculation on the probability distribution according to the value vectors to obtain the target weight sum, obtain the motion function rehabilitation evaluation prediction result according to the target weight sum, calculate the loss according to the motion function rehabilitation evaluation prediction result and the true label sequence corresponding to the training samples to obtain the loss value, and correct the internal parameters of the multi-modal data rehabilitation training model according to the loss value; Input the next group of training samples into the multi-modal data rehabilitation training model until the training situation of the multi-modal data rehabilitation training model meets the preset conditions to obtain the multi-modal data rehabilitation prediction model.

7. The method for predicting the rehabilitation of motor function based on multimodal data according to claim 6, wherein The calculation formula of the loss value is: ； wherein, is the loss value, is the total number of cycles of the input signal, is the number of cycles, is the prediction result of the motion function rehabilitation assessment, is the true label sequence corresponding to the training sample.

8. A motion function rehabilitation prediction system based on multimodal data, characterized in that, The motion function rehabilitation prediction system based on multi-modal data includes: A signal acquisition module, configured to acquire electroencephalogram signals, electromyogram signals, and functional near-infrared spectroscopy signals of a target user, and perform preprocessing on the electroencephalogram signals, the electromyogram signals, and the functional near-infrared spectroscopy signals to obtain target electroencephalogram signals, target electromyogram signals, and target functional near-infrared spectroscopy signals; A feature extraction module, configured to respectively extract features from the target electroencephalogram signals, the target electromyogram signals, and the target functional near-infrared spectroscopy signals to obtain first time-frequency features, second time-frequency features, and third time-frequency features, and perform source localization processing on the target electroencephalogram signals and the target functional near-infrared spectroscopy signals to obtain brain source intensity; A result prediction module, configured to fuse the first time-frequency features, the second time-frequency features, and the third time-frequency features to obtain target time-frequency features, and input the target time-frequency features and the brain source intensity into the trained multi-modal data rehabilitation prediction model to output the motion function rehabilitation prediction result of the target user.

9. A terminal, characterized in that, The terminal includes a memory, a processor, and a program stored on the memory and executable on the processor. When the program is executed by the processor, it implements the steps of the method for predicting motor function rehabilitation based on multimodal data according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, A computer program is stored thereon. The computer-readable storage medium stores a program for predicting motor function rehabilitation based on multimodal data. When the program for predicting motor function rehabilitation based on multimodal data is executed by a processor, it implements the steps of the method for predicting motor function rehabilitation based on multimodal data according to any one of claims 1-7.

Images