Motor function rehabilitation prediction method, system, terminal and storage medium based on multimodal data

Abstract

The present invention relates to the field of rehabilitation data prediction technology, and discloses a motor function rehabilitation prediction method, system, terminal, and storage medium based on multimodal data. The method comprises: obtaining a target user's EEG signal, electromyographic signal, and functional near-infrared spectral signal, and performing preprocessing to obtain a target EEG signal, a target electromyographic signal, and a target functional near-infrared spectral signal; performing feature extraction to obtain three time-frequency features, and performing source localization processing on the target EEG signal and the target functional near-infrared spectral signal to obtain brain source intensity; fusing the three time-frequency features to obtain a target time-frequency feature, and inputting the target time-frequency feature and brain source intensity into a multimodal data rehabilitation prediction model to output a motor function rehabilitation prediction result. The present invention models the relationship between the target user's three signals and their motor function in time series to quickly evaluate the long-term evolution of motor function in time series, thereby improving the accuracy of the prediction result.

Description

Technical Field

[0001] The present invention relates to the technical field of rehabilitation data prediction, and in particular to a motor function rehabilitation prediction method, system, terminal and storage medium based on multimodal data. Background Art

[0002] The assessment of human motor function is an important topic in the fields of rehabilitation medicine and sports science. Currently, there are mainly the following methods for motor function assessment: clinical assessment, which includes observing the patient's motor function, muscle strength, etc. However, it mainly relies on the doctor's experience and observation, which may be affected by subjective factors and makes it difficult to quantitatively assess the patient's motor function; neuroelectrophysiological assessment, although it can provide objective data, may have certain requirements for the patient's comfort and safety; motor function assessment scales, although they can provide detailed motor function scores, may not be able to comprehensively assess all aspects of the patient's motor function; robot-assisted assessment, which can provide accurate measurement of movement trajectory and muscle activity, may be costly and have high requirements for patients and operators; among the above assessment methods, clinical assessment, scale assessment, and self-assessment are all qualitative assessments, which are easily affected by the subjective state of the doctor or patient, resulting in inaccurate assessments; while neuroelectrophysiological assessment and robot-assisted assessment are quantitative assessments, they have the defects of being single modal and not containing neural pathway information.

[0003] Therefore, existing evaluation methods are mainly single-modal or dual-modal, lacking the application of functional near-infrared spectral signals and their combined use with multiple signals. In addition, most related applications lack the capture of long-term correlations between signal time series, making it impossible to effectively evaluate and predict the patient's human motor function, resulting in inaccurate prediction results of human motor function.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0005] The main purpose of the present invention is to provide a motor function rehabilitation prediction method, system, terminal and storage medium based on multimodal data, aiming to solve the problem that the existing technology lacks the application of functional near-infrared spectral signals and their combined use with multiple signals, and lacks the capture of long-term correlations between signal time series, thereby being unable to effectively evaluate and predict the patient's human motor function, resulting in inaccurate prediction results of human motor function.

[0006] To achieve the above object, the present invention provides a method for predicting motor function rehabilitation based on multimodal data, the method comprising the following steps:

[0007] Acquiring an EEG signal, an EMG signal, and a functional near-infrared spectroscopy signal of a target user, and preprocessing the EEG signal, the EMG signal, and the functional near-infrared spectroscopy signal to obtain a target EEG signal, a target EMG signal, and a target functional near-infrared spectroscopy signal;

[0008] performing feature extraction on the target EEG signal, the target EMG signal, and the target functional near-infrared spectroscopy signal to obtain a first time-frequency feature, a second time-frequency feature, and a third time-frequency feature, and performing source localization processing on the target EEG signal and the target functional near-infrared spectroscopy signal to obtain brain source intensity;

[0009] The first time-frequency feature, the second time-frequency feature and the third time-frequency feature are fused to obtain a target time-frequency feature, and the target time-frequency feature and the brain source intensity are input into a trained multimodal data rehabilitation prediction model to output the motor function rehabilitation prediction result of the target user.

[0010] Optionally, the motor function rehabilitation prediction method based on multimodal data, wherein the step of acquiring the target user's EEG signal, EMG signal, and functional near-infrared spectroscopy signal, and preprocessing the EEG signal, EMG signal, and functional near-infrared spectroscopy signal to obtain a target EEG signal, a target EMG signal, and a target functional near-infrared spectroscopy signal, specifically includes:

[0011] The experimental paradigm was set according to the clinical assessment scale, and the first sampling frequency, second sampling frequency, and third sampling frequency were set;

[0012] collecting the target user's EEG signal, EMG signal, and functional near-infrared spectroscopy signal within a preset period according to the experimental paradigm, the first sampling frequency, the second sampling frequency, and the third sampling frequency;

[0013] The EEG signal and the EMG signal are subjected to filtering processing, artifact removal and data segmentation to obtain target EEG signal and target EMG signal, and the functional near-infrared spectral signal is subjected to filtering processing and conversion processing to obtain target functional near-infrared spectral signal.

[0014] Optionally, the motor function rehabilitation prediction method based on multimodal data, wherein the feature extraction of the target EEG signal, the target EMG signal and the target functional near-infrared spectroscopy signal is performed to obtain a first time-frequency feature, a second time-frequency feature and a third time-frequency feature, specifically includes:

[0015] Performing a signal difference on the target electromyographic signal to obtain an amplitude difference, performing channel selection based on the amplitude difference to obtain a first channel, and performing feature extraction on the target electromyographic signal based on the first channel to obtain a second time-frequency feature;

[0016] Performing channel selection on the target EEG signal and the target functional near-infrared spectroscopy signal respectively through a linear model to obtain a second channel and a third channel;

[0017] Feature extraction is performed on the target EEG signal according to the second channel to obtain a first time-frequency feature, and feature extraction is performed on the target functional near-infrared spectroscopy signal according to the third channel to obtain a third time-frequency feature.

[0018] Optionally, in the motor function rehabilitation prediction method based on multimodal data, the linear model is expressed as follows:

[0019] ;

[0020] in, For measurement data matrix, is the number of data points, M is the number of channels, for Design matrix, L is the number of conditions, is the regression coefficient matrix to be estimated, is the residual error matrix.

[0021] Optionally, the motor function rehabilitation prediction method based on multimodal data, wherein the source localization processing of the target EEG signal and the target functional near-infrared spectroscopy signal to obtain brain source intensity specifically includes:

[0022] Performing current estimation on the target EEG signal to obtain a likelihood function, and determining an inverse variance mean parameter of the EEG current source according to the target functional near-infrared spectroscopy signal;

[0023] Obtaining an inverse variance parameter, setting a priori probability distribution of the inverse variance parameter according to an inverse variance mean parameter of the brain current source, and determining a priori probability distribution of the source current according to the priori probability distribution of the inverse variance parameter;

[0024] Obtaining a posterior distribution according to the prior probability distribution of the source current and the likelihood function, and performing variational inference processing on the posterior distribution to obtain a variational distribution;

[0025] The lower bound of the variational distribution is maximized to obtain a target lower bound, and data positioning is performed on the variational distribution according to the target lower bound to obtain brain source intensity.

[0026] Optionally, in the motor function rehabilitation prediction method based on multimodal data, the training process of the multimodal data rehabilitation prediction model specifically includes:

[0027] Acquire a training data set, wherein the training data set includes multiple groups of training samples, each group of training samples includes time-frequency features of historical EEG signals, time-frequency features of historical EMG signals, time-frequency features of historical functional near-infrared spectroscopy signals, and historical brain source intensity;

[0028] Creating a multimodal data rehabilitation training model, inputting a set of training samples into the multimodal data rehabilitation training model, and performing embedding processing on the training samples to obtain target training samples;

[0029] performing a linear transformation on the target training sample to obtain a query vector, a key vector, and a value vector, performing a dot product calculation on the query vector and the key vector to obtain a first dot product and a second dot product, and normalizing the first dot product and the second dot product to obtain a probability distribution;

[0030] Performing a weighted calculation on the probability distribution according to the value vector to obtain a target weight sum, obtaining a motor function rehabilitation assessment prediction result according to the target weight sum, performing a loss calculation according to the motor function rehabilitation assessment prediction result and a true label sequence corresponding to the training sample to obtain a loss value, and correcting internal parameters of the multimodal data rehabilitation training model according to the loss value;

[0031] The next set of training samples is input into the multimodal data rehabilitation training model until the training status of the multimodal data rehabilitation training model meets the preset conditions, thereby obtaining the multimodal data rehabilitation prediction model.

[0032] Optionally, in the motor function rehabilitation prediction method based on multimodal data, the calculation formula of the loss value is:

[0033] ;

[0034] in, is the loss value, is the total number of cycles of the input signal, is the number of cycles, To evaluate and predict the results of motor rehabilitation, is the true label sequence corresponding to the training sample.

[0035] Optionally, the motor function rehabilitation prediction method based on multimodal data, wherein the motor function rehabilitation prediction system based on multimodal data includes:

[0036] a signal acquisition module, configured to acquire the target user's EEG signals, EMG signals, and functional near-infrared spectroscopy signals, and pre-process the EEG signals, EMG signals, and functional near-infrared spectroscopy signals to obtain target EEG signals, target EMG signals, and target functional near-infrared spectroscopy signals;

[0037] a feature extraction module, configured to perform feature extraction on the target EEG signal, the target EMG signal, and the target functional near-infrared spectroscopy signal, respectively, to obtain a first time-frequency feature, a second time-frequency feature, and a third time-frequency feature, and to perform source localization processing on the target EEG signal and the target functional near-infrared spectroscopy signal to obtain brain source intensity;

[0038] The result prediction module is used to perform feature fusion on the first time-frequency feature, the second time-frequency feature and the third time-frequency feature to obtain the target time-frequency feature, and input the target time-frequency feature and the brain source intensity into the trained multimodal data rehabilitation prediction model to output the motor function rehabilitation prediction result of the target user.

[0039] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a program stored on the memory and runnable on the processor, and when the program is executed by the processor, the steps of the motor function rehabilitation prediction method based on multimodal data as described above are implemented.

[0040] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a motor function rehabilitation prediction program based on multimodal data, and when the motor function rehabilitation prediction program based on multimodal data is executed by a processor, the steps of the motor function rehabilitation prediction method based on multimodal data as described above are implemented.

[0041] In the present invention, the EEG signal, EMG signal and functional near-infrared spectral signal of the target user are obtained, and the EEG signal, EMG signal and functional near-infrared spectral signal are preprocessed to obtain target EEG signal, target EMG signal and target functional near-infrared spectral signal; feature extraction is performed on the target EEG signal, target EMG signal and target functional near-infrared spectral signal respectively to obtain a first time-frequency feature, a second time-frequency feature and a third time-frequency feature, and source localization processing is performed on the target EEG signal and the target functional near-infrared spectral signal to obtain brain source intensity; feature fusion is performed on the first time-frequency feature, the second time-frequency feature and the third time-frequency feature to obtain a target time-frequency feature, and the target time-frequency feature and the brain source intensity are input into a trained multimodal data rehabilitation prediction model to output the motor function rehabilitation prediction result of the target user. The present invention mines multiple information to construct a multimodal data rehabilitation prediction model for the motor function of people with movement disorders, and models the relationship between the target user's EEG signals, EMG signals and functional near-infrared spectroscopy signals and their motor function in time series to quickly evaluate the long-term evolution of motor function in time series, thereby improving the accuracy of the prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flowchart of a preferred embodiment of the motor function rehabilitation prediction method based on multimodal data in the present invention;

[0043] Figure 2 Schematic diagram of the overall process of the motor function rehabilitation prediction method based on multimodal data in the present invention;

[0044] Figure 3 Schematic diagram of the network structure of the multimodal data rehabilitation training model of a preferred embodiment of the present invention;

[0045] Figure 4 It is a structural diagram of a preferred embodiment of the motor function rehabilitation prediction system based on multimodal data in the present invention;

[0046] Figure 5 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0048] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), such directional indications are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0049] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features specified as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that ordinary technicians in this field can implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0050] The motor function rehabilitation prediction method based on multimodal data described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the motor function rehabilitation prediction method based on multimodal data includes the following steps:

[0051] Step S10: Obtain the EEG signal, EMG signal, and functional near-infrared spectroscopy signal of the target user, and pre-process the EEG signal, EMG signal, and functional near-infrared spectroscopy signal to obtain a target EEG signal, a target EMG signal, and a target functional near-infrared spectroscopy signal.

[0052] The step S10 includes:

[0053] Step S11: setting an experimental paradigm according to a clinical assessment scale, and setting a first sampling frequency, a second sampling frequency, and a third sampling frequency;

[0054] Step S12: collecting the target user's EEG signals, EMG signals, and functional near-infrared spectroscopy signals within a preset period according to the experimental paradigm, the first sampling frequency, the second sampling frequency, and the third sampling frequency;

[0055] Step S13: filtering, removing artifacts, and data segmenting the EEG signal and the EMG signal to obtain target EEG signal and target EMG signal; filtering and converting the functional near-infrared spectral signal to obtain a target functional near-infrared spectral signal.

[0056] Specifically, existing methods are mainly single-modal or dual-modal, lacking the application of functional near-infrared spectroscopy signals and their integrated use with multiple signals. Most of them only use the common amplitude and frequency characteristics of the signals, lacking in-depth mining and use of the time and space information of the signals, and neural networks lack the ability to capture the long-term correlation between signal time series in most related applications. The present invention will comprehensively use synchronously collected EEG signals, EMG signals and functional near-infrared spectroscopy signals for comprehensive evaluation and prediction, extract time-frequency features from signals selected through key channels to help the model learn quickly, perform comprehensive source localization on all-channel brain signals to supplement the information missing due to feature extraction, and use neural networks to autonomously learn two types of signal features, thereby achieving rehabilitation assessment and prediction of motor function of people with disabilities; in an embodiment of the present invention, by synchronously collecting EMG signals, EMG signals and functional near-infrared spectroscopy signals of target users (e.g., people with movement disorders) under a specified experimental paradigm at specific periodic intervals (e.g., every other week), and synchronously recording the motor function score results obtained by the physician according to the scale of the target user's evaluation, i.e., the motor function score. After that, data preprocessing, channel selection, feature extraction and source localization are performed on the electromyographic signals, electroencephalographic signals and functional near-infrared spectroscopy signals. Then, a model is established based on the attention mechanism network, and motion evaluation is performed based on the fused feature array. The specific process is as follows: Figure 2 shown.

[0057] First, the experimental paradigm is set according to the clinical assessment scale, wherein the experimental paradigm includes the upper limb movement paradigm, the lower limb movement paradigm, the location and duration of myoelectric signal acquisition, the location and duration of EEG signal acquisition, the location and duration of functional near-infrared spectroscopy signal acquisition, the clinical assessment cycle and the experimental acquisition cycle; the experimental paradigm is carried out periodically (for example, every other week), the scoring results obtained by the physician based on the scale for the target user are recorded, and the first sampling frequency (i.e., the sampling frequency of the EEG signal, the commonly used sampling frequencies are 250Hz, 500Hz or 1000Hz), the second sampling frequency (i.e., the sampling frequency of the EMG signal, the commonly used sampling frequencies are 250Hz, 500Hz or 1000Hz), and the second sampling frequency (i.e., the sampling frequency of the EMG signal). The first sampling frequency, the second sampling frequency and the third sampling frequency are commonly used (500 Hz or 1000 Hz) and the third sampling frequency (i.e., the sampling frequency of the functional near-infrared spectroscopy signal, the commonly used sampling frequency is 10 Hz). The EEG signal, EMG signal and functional near-infrared spectroscopy signal of the target user are synchronously collected according to the first sampling frequency, the second sampling frequency and the third sampling frequency. However, during the signal collection process, many interference signals may be encountered, such as electrooculogram signals, heartbeat artifacts and interference from the industrial frequency power supply. These interferences may reduce the signal-to-noise ratio of the signal, thereby negatively affecting the accuracy of subsequent feature extraction and prediction evaluation. Therefore, it is necessary to preprocess the synchronously collected EEG signals, EMG signals and functional near-infrared spectroscopy signals to obtain target EEG signals, target EMG signals and target functional near-infrared spectroscopy signals. Specifically, for the EEG signals and EMG signals, the main steps are filtering, artifact removal and data segmentation; and for the functional near-infrared spectroscopy signals, filtering is first performed to eliminate physiological noise, and then the improved Beer-Lambert law is used to convert the changes in the original optical density signal into the optical density of each time point. Changes in oxygenated hemoglobin concentration and changes in deoxyhemoglobin concentration , the corresponding expression is:

[0058] ;

[0059] in, and Oxygenated hemoglobin at wavelengths and The absorption coefficient under and Deoxyhemoglobin at wavelength and The absorption coefficient under and The wavelengths are and The optical density change under , and the calculation formula of optical density change is: ;in, For the wavelength The optical density changes under For the wavelength The reference signal under For the wavelength The following detection signal.

[0060] Step S20: performing feature extraction on the target EEG signal, the target EMG signal, and the target functional near-infrared spectral signal, respectively, to obtain a first time-frequency feature, a second time-frequency feature, and a third time-frequency feature; and performing source localization processing on the target EEG signal and the target functional near-infrared spectral signal to obtain brain source intensity.

[0061] Specifically, after obtaining the target EEG signal, target EMG signal, and target functional near-infrared spectroscopy signal, it is necessary to select channels according to different rules to eliminate redundant or low-quality information to ensure the accuracy of the evaluation system. The specific process is as follows: for the target EMG signal, subtract the minimum value of the target EMG signal from the median value of each electrode signal of the target user to obtain the amplitude difference; take the average of the amplitude differences of all muscle electrodes of the target user, and use the minimum mean as the threshold to select the channel to obtain the first channel. For the target EEG signal and the target functional near-infrared spectroscopy signal, a linear model is used to select the channel to obtain the second and third channels, wherein the expression of the linear model is:

[0062] ;

[0063] in, For measurement data matrix, is the number of data points, M is the number of channels, for Design matrix, L is the number of conditions, including tasks and any items considered to be sources of variance in the data, is the regression coefficient matrix to be estimated, is the residual error matrix.

[0064] For signals passing through a single sample Inspection to test and , to identify values, thus representing channels with significant contrast between motor execution tasks. The value calculation formula is:

[0065] ;

[0066] in, c is a contrast vector that determines the contrast between specific conditions, is the total number of cycles of the input signal. The linear model is used to estimate the value of each individual functional near-infrared spectral channel (i.e., the third channel) , select a group from The channels with significant contrast between the motor execution tasks are selected as candidate channels; for the left and right brain regions, the channels with higher The functional near-infrared spectroscopy channel of the spectral image is represented by the value of the spectral function, and its adjacent EEG channel (i.e., the second channel).

[0067] Afterwards, the processed data is divided into fixed time window lengths (e.g., 10ms, 50ms, etc.) to extract features, obtaining the first time-frequency feature, the second time-frequency feature, and the third time-frequency feature. For the first time-frequency feature and the second time-frequency feature, the target electromyographic signal of the first channel and the target electroencephalographic signal of the second channel are extracted within the time window length. RMS and the mean absolute value MAV , the corresponding extraction formula is:

[0068] ;

[0069] ;

[0070] in, N is the total number of data points in a fixed time window, The signal at time Then, the target EEG signal and target EMG signal are Fourier transformed to calculate the power spectral density , then calculate the eigenvalue median frequency MF and average power frequency MPF , the corresponding calculation formula is:

[0071] ;

[0072] ;

[0073] in, is the frequency value, is the highest frequency of the signal. Similarly, the mean value of the concentration change signal of oxygenated hemoglobin and deoxygenated hemoglobin converted by functional near-infrared spectroscopy is extracted SM , slope SLP , kurtosis KRT , skewness SKW The corresponding extraction formula is:

[0074] ;

[0075] ;

[0076] ;

[0077] ;

[0078] in, The signal at time N The value at is the mean value of the signal in this time window, is the standard deviation of the signal in this time window.

[0079] Subsequently, comprehensive source localization is performed on all-channel brain signals to supplement the information missing due to feature extraction and obtain more comprehensive information. Specifically, multimodal source localization is performed on the target EEG signal and the target functional near-infrared spectroscopy signal under the variational Bayesian framework to achieve deep signal mining. is the current density of the source activity on the cerebral cortex (i.e., source intensity). The current source distribution model is used to locate the source of the target EEG signal. The corresponding expression is:

[0080] ;

[0081] in, is the scalp EEG signal, G is the lead field matrix, i.e. the spatial relationship between the scalp EEG signal and the source current, is the noise term, which is set to obey a Gaussian distribution with a mean of zero and a spherical covariance. Hierarchical Bayesian is used to estimate the current of the target EEG signal to obtain the likelihood function , expressed as:

[0082] ;

[0083] in, is the inverse of the EEG noise variance. And for the prior probability distribution of source current First set a normal prior, expressed as:

[0084] ;

[0085] in, for The transpose of is the inverse variance parameter of the unknown brain current source, For its diagonal matrix, set the prior probability distribution of the inverse variance parameter for:

[0086] ;

[0087] in, The mean is , the degrees of freedom are The distribution of Gamma distribution, is set to 10, and for unknown The parameters (i.e., the inverse variance mean parameters of the brain current source) are determined using the target functional near-infrared spectroscopy data, and the corresponding expression is:

[0088] ;

[0089] in, is the baseline of the previous current variance, obtained by using the minimum norm estimation from the baseline interval of the EEG data, The target functional near infrared spectroscopy data Normalized activity data at the moment, is the variance magnification factor, usually set to 100.

[0090] After determining the prior distribution, we maximize the following posterior distribution Find the best result, the expression is:

[0091] ;

[0092] in, is the likelihood function, that is, given the source intensity When EEG data The probability of occurrence, is the prior distribution of source current, EEG data The marginal probability of . It is difficult to calculate directly, so the variational inference method is introduced here to approximate its posterior distribution as a parameterized variational distribution , and by maximizing the lower bound of the variational distribution To approximate the true posterior, the corresponding formula is:

[0093] ;

[0094] in, For EEG data E With source strength The logarithmic expectation of the joint probability, is the entropy of the variational distribution, which is used to prevent overfitting. Subsequently, the lower bound of the variational distribution is optimized by the logarithmic expectation maximization method. Specifically, other parameters are fixed and Maximize to get the optimal variational distribution , the calculation formula is:

[0095] ;

[0096] After that, fix the optimal variational distribution optimization about Estimates of brain source intensity , the corresponding expression is:

[0097] .

[0098] Step S30: Fusing the first time-frequency feature, the second time-frequency feature, and the third time-frequency feature to obtain a target time-frequency feature, and inputting the target time-frequency feature and the brain source intensity into a trained multimodal data rehabilitation prediction model to output the motor function rehabilitation prediction result of the target user.

[0099] Specifically, in an embodiment of the present invention, the first time-frequency feature, the second time-frequency feature and the third time-frequency feature are fused to obtain a target time-frequency feature. Then, the target time-frequency feature and the brain source intensity are input into a trained multimodal data rehabilitation prediction model. Before inputting the multimodal data rehabilitation prediction model, it is necessary to train a multimodal data rehabilitation training model (i.e., a Transformer Encoder network model) to obtain a multimodal data rehabilitation prediction model. The network structure of the multimodal data rehabilitation training model is as follows: Figure 3 As shown in Figure 2, this multimodal data rehabilitation training model uses a parallel network architecture: First, the model embeds the raw feature input data at each time step, mapping it to a higher-dimensional space using a linear layer, making it more suitable for learning. Subsequently, the model adds a positional encoding corresponding to the input time step to each data point, representing the temporal order, allowing the model to learn temporal features. With the existence of position encoding, the feature sequence of each time step can be input in parallel into the same network model to participate in the calculation together, which can not only improve the training speed of the algorithm, but also make up for the deficiency of the traditional network model in forgetting early information, capture long-term dependencies, and enhance the expression ability of the model. The specific training process is to obtain a training data set, wherein the training data set includes multiple groups of training samples, each group of the training samples includes the time-frequency characteristics of historical EEG signals, the time-frequency characteristics of historical electromyographic signals, the time-frequency characteristics of historical functional near-infrared spectral signals and historical brain source intensity; a group of training samples are input into the multimodal data rehabilitation training model, and the training samples are embedded through the encoder to obtain target training samples.

[0100] For the encoder part, it is composed of n identical layers stacked together, usually 2-6 layers. When calculating the self-attention of the encoder, the data comes from the output of the previous encoder layer. Each encoder layer has two sub-layers, namely multi-head self-attention convergence and position-based feedforward network. Each sub-layer adopts residual connection, so that the input can be propagated forward faster through the cross-layer data line. This also requires that for any input x at any position in the sequence, the sub-layer output sublayer(x) must be satisfied for the calculation of the residual connection. Therefore, for each position corresponding to the input sequence, the encoder will output a d-dimensional representation vector. After the addition calculation of the residual connection, the model immediately applies layer normalization, and normalization is performed based on the feature dimension to achieve magnitude uniformity and improve accuracy. Self-attention is the core mechanism of Transformer. Its purpose is to capture dependencies between each time step of the input. The feature of each time step of the input is a vector , the main steps of self-attention include: for each time step input, the model will obtain three vectors through linear transformation: query vector , key vector Sum value vector , where the three weight matrices of the linear transformation are trainable parameters, that is, the target training sample is linearly transformed to obtain the query vector, key vector and value vector; then, for each time step, calculate and The first dot product and the second dot product are obtained, and the score is converted into a probability distribution by normalization, which reflects the strength of the relationship between the two time steps, that is, the first dot product and the second dot product are normalized to obtain a probability distribution. Finally, each The weighted sum obtained by weighted calculation of the probability distribution is used as the output of self-attention, which represents the contextual information of the time step combined with all other time steps. The sub-layer multi-head attention uses multiple attention heads to focus on different relationships and patterns at the same time. Each attention head learns different query mappings, key mappings, and value mappings, and calculates multiple different weighted sum values; the outputs of all attention heads are spliced ​​together and linearly transformed to obtain the final output, which is the prediction result of motor function rehabilitation assessment, and is used express, , the number of attention heads here can be set to 3-8.

[0101] The self-attention layer captures the relationship between different time steps, but the features of each time step still need to be further transformed nonlinearly through the feedforward network to improve the model's expressiveness. The position-based feedforward network uses the same multi-layer perceptron to transform the representation of all positions in the sequence, which is why the feedforward network is called position-based. Afterwards, based on the motor function rehabilitation assessment prediction results and the true label sequence corresponding to the training sample (i.e. , ) to calculate the loss and obtain the loss value, wherein the calculation formula of the loss value is: ; is the loss value, is the total number of cycles of the input signal; thereafter, the internal parameters of the multimodal data rehabilitation training model are corrected according to the loss value; the next set of training samples is input into the multimodal data rehabilitation training model until the training status of the multimodal data rehabilitation training model meets the preset conditions, thereby obtaining the multimodal data rehabilitation prediction model.

[0102] After obtaining the trained multimodal data rehabilitation prediction model, the target time-frequency features and the brain source intensity are input into the trained multimodal data rehabilitation prediction model, and the motor function rehabilitation prediction result of the target user is output, thereby effectively evaluating and predicting the temporal evolution of the target user's human motor function rehabilitation situation, so as to improve the accuracy of the target user's motor function prediction. The present invention mines multiple information to construct a multimodal data rehabilitation prediction model for the motor function of people with movement disorders, and models the relationship between the target user's EEG signals, electromyographic signals, and functional near-infrared spectroscopy signals and their motor function in temporal order, so as to quickly evaluate the long-term temporal evolution of the motor function, thereby improving the accuracy of the prediction results.

[0103] Further, if Figure 4 As shown, based on the above-mentioned motor function rehabilitation prediction method based on multimodal data, the present invention also provides a motor function rehabilitation prediction system based on multimodal data, and the motor function rehabilitation prediction system based on multimodal data includes:

[0104] a signal acquisition module 51 for acquiring an EEG signal, an EMG signal, and a functional near-infrared spectroscopy signal of a target user, and preprocessing the EEG signal, the EMG signal, and the functional near-infrared spectroscopy signal to obtain a target EEG signal, a target EMG signal, and a target functional near-infrared spectroscopy signal;

[0105] a feature extraction module 52 for performing feature extraction on the target EEG signal, the target EMG signal, and the target functional near-infrared spectroscopy signal, respectively, to obtain a first time-frequency feature, a second time-frequency feature, and a third time-frequency feature, and performing source localization processing on the target EEG signal and the target functional near-infrared spectroscopy signal to obtain brain source intensity;

[0106] The result prediction module 53 is used to perform feature fusion on the first time-frequency feature, the second time-frequency feature and the third time-frequency feature to obtain the target time-frequency feature, and input the target time-frequency feature and the brain source intensity into the trained multimodal data rehabilitation prediction model to output the motor function rehabilitation prediction result of the target user.

[0107] Further, if Figure 5 As shown, based on the above-mentioned motor function rehabilitation prediction method based on multimodal data, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 5 Only some of the components of the terminal are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0108] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a motor function rehabilitation prediction program 40 based on multimodal data is stored on the memory 20, and the motor function rehabilitation prediction program 40 based on multimodal data can be executed by the processor 10, thereby realizing the motor function rehabilitation prediction method based on multimodal data in the present application.

[0109] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program codes or process data stored in the memory 20, such as executing the motor function rehabilitation prediction method based on multimodal data.

[0110] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch screen, etc. The display 30 is used to display information on the terminal and to display a visual user interface.

[0111] In one embodiment, when the processor 10 executes the program 40 for predicting motor function rehabilitation based on multimodal data in the memory 20, the following steps are implemented:

[0112] Acquiring an EEG signal, an EMG signal, and a functional near-infrared spectroscopy signal of a target user, and preprocessing the EEG signal, the EMG signal, and the functional near-infrared spectroscopy signal to obtain a target EEG signal, a target EMG signal, and a target functional near-infrared spectroscopy signal;

[0113] performing feature extraction on the target EEG signal, the target EMG signal, and the target functional near-infrared spectroscopy signal to obtain a first time-frequency feature, a second time-frequency feature, and a third time-frequency feature, and performing source localization processing on the target EEG signal and the target functional near-infrared spectroscopy signal to obtain brain source intensity;

[0114] The first time-frequency feature, the second time-frequency feature and the third time-frequency feature are fused to obtain a target time-frequency feature, and the target time-frequency feature and the brain source intensity are input into a trained multimodal data rehabilitation prediction model to output the motor function rehabilitation prediction result of the target user.

[0115] The step of obtaining the target user's EEG signal, EMG signal, and functional near-infrared spectroscopy signal, and preprocessing the EEG signal, EMG signal, and functional near-infrared spectroscopy signal to obtain a target EEG signal, a target EMG signal, and a target functional near-infrared spectroscopy signal specifically includes:

[0116] The experimental paradigm was set according to the clinical assessment scale, and the first sampling frequency, second sampling frequency, and third sampling frequency were set;

[0117] collecting the target user's EEG signal, EMG signal, and functional near-infrared spectroscopy signal within a preset period according to the experimental paradigm, the first sampling frequency, the second sampling frequency, and the third sampling frequency;

[0118] The EEG signal and the EMG signal are subjected to filtering processing, artifact removal and data segmentation to obtain target EEG signal and target EMG signal, and the functional near-infrared spectral signal is subjected to filtering processing and conversion processing to obtain target functional near-infrared spectral signal.

[0119] The step of extracting features from the target EEG signal, the target EMG signal, and the target functional near-infrared spectroscopy signal to obtain a first time-frequency feature, a second time-frequency feature, and a third time-frequency feature specifically includes:

[0120] Performing a signal difference on the target electromyographic signal to obtain an amplitude difference, performing channel selection based on the amplitude difference to obtain a first channel, and performing feature extraction on the target electromyographic signal based on the first channel to obtain a second time-frequency feature;

[0121] Performing channel selection on the target EEG signal and the target functional near-infrared spectroscopy signal respectively through a linear model to obtain a second channel and a third channel;

[0122] Feature extraction is performed on the target EEG signal according to the second channel to obtain a first time-frequency feature, and feature extraction is performed on the target functional near-infrared spectroscopy signal according to the third channel to obtain a third time-frequency feature.

[0123] Wherein, the expression of the linear model is:

[0124] ;

[0125] in, For measurement data matrix, is the number of data points, M is the number of channels, for Design matrix, L is the number of conditions, is the regression coefficient matrix to be estimated, is the residual error matrix.

[0126] The performing source localization processing on the target EEG signal and the target functional near-infrared spectroscopy signal to obtain brain source intensity specifically includes:

[0127] Performing current estimation on the target EEG signal to obtain a likelihood function, and determining an inverse variance mean parameter of an EEG current source according to the target functional near-infrared spectroscopy signal;

[0128] Obtaining an inverse variance parameter, setting a priori probability distribution of the inverse variance parameter according to an inverse variance mean parameter of the brain current source, and determining a priori probability distribution of the source current according to the priori probability distribution of the inverse variance parameter;

[0129] Obtaining a posterior distribution according to the prior probability distribution of the source current and the likelihood function, and performing variational inference processing on the posterior distribution to obtain a variational distribution;

[0130] The lower bound of the variational distribution is maximized to obtain a target lower bound, and data positioning is performed on the variational distribution according to the target lower bound to obtain brain source intensity.

[0131] The training process of the multimodal data rehabilitation prediction model specifically includes:

[0132] Acquire a training data set, wherein the training data set includes multiple groups of training samples, each group of training samples includes time-frequency features of historical EEG signals, time-frequency features of historical EMG signals, time-frequency features of historical functional near-infrared spectroscopy signals, and historical brain source intensity;

[0133] Creating a multimodal data rehabilitation training model, inputting a set of training samples into the multimodal data rehabilitation training model, and performing embedding processing on the training samples to obtain target training samples;

[0134] performing a linear transformation on the target training sample to obtain a query vector, a key vector, and a value vector, performing a dot product calculation on the query vector and the key vector to obtain a first dot product and a second dot product, and normalizing the first dot product and the second dot product to obtain a probability distribution;

[0135] Performing a weighted calculation on the probability distribution according to the value vector to obtain a target weight sum, obtaining a motor function rehabilitation assessment prediction result according to the target weight sum, performing a loss calculation according to the motor function rehabilitation assessment prediction result and a true label sequence corresponding to the training sample to obtain a loss value, and correcting internal parameters of the multimodal data rehabilitation training model according to the loss value;

[0136] The next set of training samples is input into the multimodal data rehabilitation training model until the training status of the multimodal data rehabilitation training model meets the preset conditions, thereby obtaining the multimodal data rehabilitation prediction model.

[0137] The calculation formula of the loss value is:

[0138] ;

[0139] in, is the loss value, is the total number of cycles of the input signal, is the number of cycles, To evaluate and predict the results of motor rehabilitation, is the true label sequence corresponding to the training sample.

[0140] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a motor function rehabilitation prediction program based on multimodal data, and when the motor function rehabilitation prediction program based on multimodal data is executed by a processor, the steps of the motor function rehabilitation prediction method based on multimodal data as described above are implemented.

[0141] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0142] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When executed, the program can include the processes in the above-described method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0143] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A motor function rehabilitation prediction method based on multimodal data, characterized in that: The motor function rehabilitation prediction method based on multimodal data includes: Acquiring an EEG signal, an EMG signal, and a functional near-infrared spectroscopy signal of a target user, and preprocessing the EEG signal, the EMG signal, and the functional near-infrared spectroscopy signal to obtain a target EEG signal, a target EMG signal, and a target functional near-infrared spectroscopy signal; performing feature extraction on the target EEG signal, the target EMG signal, and the target functional near-infrared spectroscopy signal to obtain a first time-frequency feature, a second time-frequency feature, and a third time-frequency feature, and performing source localization processing on the target EEG signal and the target functional near-infrared spectroscopy signal to obtain brain source intensity; The performing source localization processing on the target EEG signal and the target functional near-infrared spectroscopy signal to obtain brain source intensity specifically includes: Performing current estimation on the target EEG signal to obtain a likelihood function, and determining an inverse variance mean parameter of an EEG current source according to the target functional near-infrared spectroscopy signal; Obtaining an inverse variance parameter, setting a priori probability distribution of the inverse variance parameter according to an inverse variance mean parameter of the brain current source, and determining a priori probability distribution of the source current according to the priori probability distribution of the inverse variance parameter; Obtaining a posterior distribution according to the prior probability distribution of the source current and the likelihood function, and performing variational inference processing on the posterior distribution to obtain a variational distribution; Maximizing the lower bound of the variational distribution to obtain a target lower bound, and performing data positioning on the variational distribution according to the target lower bound to obtain brain source intensity; Performing feature fusion on the first time-frequency feature, the second time-frequency feature, and the third time-frequency feature to obtain a target time-frequency feature, and inputting the target time-frequency feature and the brain source intensity into a trained multimodal data rehabilitation prediction model to output a motor function rehabilitation prediction result for the target user; The training process of the multimodal data rehabilitation prediction model specifically includes: Acquire a training data set, wherein the training data set includes multiple groups of training samples, each group of training samples includes time-frequency features of historical EEG signals, time-frequency features of historical EMG signals, time-frequency features of historical functional near-infrared spectroscopy signals, and historical brain source intensity; Creating a multimodal data rehabilitation training model, inputting a set of training samples into the multimodal data rehabilitation training model, and performing embedding processing on the training samples to obtain target training samples; performing a linear transformation on the target training sample to obtain a query vector, a key vector, and a value vector, performing a dot product calculation on the query vector and the key vector to obtain a first dot product and a second dot product, and normalizing the first dot product and the second dot product to obtain a probability distribution; Performing a weighted calculation on the probability distribution according to the value vector to obtain a target weight sum, obtaining a motor function rehabilitation assessment prediction result according to the target weight sum, performing a loss calculation according to the motor function rehabilitation assessment prediction result and a true label sequence corresponding to the training sample to obtain a loss value, and correcting internal parameters of the multimodal data rehabilitation training model according to the loss value; Inputting the next set of training samples into the multimodal data rehabilitation training model until the training status of the multimodal data rehabilitation training model meets the preset conditions, thereby obtaining the multimodal data rehabilitation prediction model; The calculation formula of the loss value is: ； in, is the loss value, is the total number of cycles of the input signal, is the number of cycles, To evaluate and predict the results of motor rehabilitation, is the true label sequence corresponding to the training sample.

2. The motor function rehabilitation prediction method based on multimodal data according to claim 1, characterized in that: The step of obtaining the target user's EEG signal, EMG signal, and functional near-infrared spectroscopy signal, and preprocessing the EEG signal, EMG signal, and functional near-infrared spectroscopy signal to obtain a target EEG signal, a target EMG signal, and a target functional near-infrared spectroscopy signal specifically includes: The experimental paradigm was set according to the clinical assessment scale, and the first sampling frequency, second sampling frequency, and third sampling frequency were set; collecting the target user's EEG signal, EMG signal, and functional near-infrared spectroscopy signal within a preset period according to the experimental paradigm, the first sampling frequency, the second sampling frequency, and the third sampling frequency; The EEG signal and the EMG signal are subjected to filtering processing, artifact removal and data segmentation to obtain target EEG signal and target EMG signal, and the functional near-infrared spectral signal is subjected to filtering processing and conversion processing to obtain target functional near-infrared spectral signal.

3. The motor function rehabilitation prediction method based on multimodal data according to claim 1, characterized in that: The feature extraction of the target EEG signal, the target EMG signal, and the target functional near-infrared spectroscopy signal is performed to obtain a first time-frequency feature, a second time-frequency feature, and a third time-frequency feature, specifically including: Performing a signal difference on the target electromyographic signal to obtain an amplitude difference, performing channel selection based on the amplitude difference to obtain a first channel, and performing feature extraction on the target electromyographic signal based on the first channel to obtain a second time-frequency feature; Performing channel selection on the target EEG signal and the target functional near-infrared spectroscopy signal respectively through a linear model to obtain a second channel and a third channel; Feature extraction is performed on the target EEG signal according to the second channel to obtain a first time-frequency feature, and feature extraction is performed on the target functional near-infrared spectroscopy signal according to the third channel to obtain a third time-frequency feature.

4. The method for predicting motor function rehabilitation based on multimodal data according to claim 3, characterized in that: The linear model is expressed as: ； in, For measurement data matrix, is the number of data points, M is the number of channels, for Design matrix, L is the number of conditions, is the regression coefficient matrix to be estimated, is the residual error matrix.

5. A motor function rehabilitation prediction system based on multimodal data, characterized in that: The motor function rehabilitation prediction system based on multimodal data is applied to the motor function rehabilitation prediction method based on multimodal data according to any one of claims 1 to 4, and the motor function rehabilitation prediction system based on multimodal data includes: a signal acquisition module, configured to acquire the target user's EEG signals, EMG signals, and functional near-infrared spectroscopy signals, and pre-process the EEG signals, EMG signals, and functional near-infrared spectroscopy signals to obtain target EEG signals, target EMG signals, and target functional near-infrared spectroscopy signals; a feature extraction module, configured to perform feature extraction on the target EEG signal, the target EMG signal, and the target functional near-infrared spectroscopy signal, respectively, to obtain a first time-frequency feature, a second time-frequency feature, and a third time-frequency feature, and to perform source localization processing on the target EEG signal and the target functional near-infrared spectroscopy signal to obtain brain source intensity; The result prediction module is used to perform feature fusion on the first time-frequency feature, the second time-frequency feature and the third time-frequency feature to obtain the target time-frequency feature, and input the target time-frequency feature and the brain source intensity into the trained multimodal data rehabilitation prediction model to output the motor function rehabilitation prediction result of the target user.

6. A terminal, characterized in that: The terminal includes a memory, a processor, and a program stored in the memory and executable on the processor. When the program is executed by the processor, the steps of the motor function rehabilitation prediction method based on multimodal data as described in any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer-readable storage medium stores a motor function rehabilitation prediction program based on multimodal data. When the motor function rehabilitation prediction program based on multimodal data is executed by a processor, the steps of the motor function rehabilitation prediction method based on multimodal data as described in any one of claims 1-4 are implemented.

Images