Motion intention decoding method and system

Abstract

The present invention discloses a method and system for decoding motion intention, wherein the method comprises the following steps: obtaining first signal data; generating corresponding prediction results based on the first signal data and a prediction model, wherein the prediction results are used to indicate the motion intention corresponding to the user's next action; generating and outputting corresponding decoding results based on the first signal data and an intention recognition model, wherein the decoding results are used to indicate the motion intention corresponding to the user's current action; calculating the degree of deviation between the decoding result and the prediction result; when the degree of deviation exceeds a preset deviation threshold, performing a reverse update on the intention recognition model based on the degree of deviation. While decoding the current action, the present invention also performs a forward-looking prediction on the next action, and automatically optimizes the intention recognition model based on the deviation between the prediction result and the decoding result, so that it is suitable for complex collaborative tasks or multi-stage tasks.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a motion intention decoding method and system. Background Art

[0002] Existing methods mainly rely on real-time collection of physiological signals for decoding, which makes it difficult to meet the needs of complex collaborative tasks or multi-stage tasks.

[0003] Physiological signals include:

[0004] Electroencephalogram (EEG);

[0005] Electromyography (EMG);

[0006] Functional Near-Infrared Spectroscopy (fNIRS);

[0007] Motion and Posture Signals (MPS). Summary of the invention

[0008] Aiming at the shortcoming that the motion intention decoding method disclosed in the prior art is not suitable for complex collaborative tasks or multi-stage tasks, the present invention proposes a motion intention decoding method and system.

[0009] In order to solve the above technical problems, the present invention is solved by the following technical solutions:

[0010] A motion intention decoding method comprises the following steps:

[0011] Acquire first signal data, where the first signal data includes one or more physiological signals corresponding to a current exercise period of the user;

[0012] Based on the first signal data and the prediction model, a corresponding prediction result is generated, where the prediction result is used to indicate the movement intention corresponding to the user's next action;

[0013] Based on the first signal data and the intention recognition model, a corresponding decoding result is generated and output, wherein the decoding result is used to indicate the movement intention corresponding to the user's current action;

[0014] Calculating the degree of deviation between the decoding result and the prediction result;

[0015] When the degree of deviation exceeds a preset deviation threshold, the intention recognition model is reversely updated based on the degree of deviation.

[0016] This implementation is applicable to scenarios based on single physiological signal decoding or multi-modal signal fusion decoding;

[0017] This embodiment makes a forward-looking prediction of the user's next action while decoding, and when the deviation between the prediction result and the decoding result is large, the deviation is used as a feedback signal to optimize the intention recognition model. Even if the difference between the previous and subsequent actions is large, the accuracy of decoding can be guaranteed.

[0018] As an implementable method:

[0019] The first signal data includes a first physiological signal and a plurality of second physiological signals, wherein the first physiological signal is an electroencephalogram signal;

[0020] The intention recognition model includes a unimodal decoding model and a multimodal decoding model. When the deviation between the corresponding decoding result and the prediction result exceeds a preset deviation threshold, the unimodal decoding model and the multimodal decoding model are reversely updated based on the deviation;

[0021] Based on the first signal data and the intention recognition model, the specific steps of generating a corresponding decoding result include:

[0022] Comparing the signal quality score corresponding to the first physiological signal with a preset first score threshold;

[0023] When the signal quality score corresponding to the first physiological signal reaches the first score threshold, decoding the first physiological signal based on the single-modality decoding model to obtain a corresponding decoding result;

[0024] When the signal quality score corresponding to the first physiological signal does not reach the first scoring threshold, the first physiological signal is used as the target decoding signal, and the second physiological signal with a signal quality score higher than the second scoring threshold is used as the target decoding signal. The multimodal decoding model performs multimodal fusion decoding based on each target decoding signal to obtain a corresponding decoding result.

[0025] This embodiment uses the EEG signal as the main decoding signal. When the signal quality of the EEG signal is good, single-modal decoding is performed based on the EEG signal. When the signal quality of the EEG signal is poor, other physiological signals with better signal quality are used as supplements to perform multi-modal decoding. This embodiment can intelligently select an appropriate signal decoding method, which can not only improve the accuracy and robustness of decoding, but also effectively improve the real-time and adaptability of the system.

[0026] As an implementable method:

[0027] The second physiological signal is an electromyographic signal, a functional near infrared signal and a posture signal.

[0028] As an implementable method:

[0029] The prediction model is a digital twin prediction model;

[0030] The input of the digital twin prediction model is EEG signals, EMG signals and posture signals, and the output is the corresponding prediction results.

[0031] As an implementable method:

[0032] The specific steps of performing multimodal fusion decoding based on each target decoding signal by the multimodal decoding model are as follows:

[0033] Extracting decoding features corresponding to each target decoding signal;

[0034] Perform feature weighting processing based on the signal quality score and decoding features corresponding to each target decoding signal to obtain corresponding fusion features;

[0035] The fusion feature is input into the multimodal decoding model, and the multimodal decoding model outputs a corresponding decoding result.

[0036] As an implementable method, feature weighting processing is performed based on the signal quality score and decoding features corresponding to each target decoded signal to obtain the corresponding fusion features in the following specific steps:

[0037] Get the decoding requirements corresponding to the current decoding task;

[0038] When the decoding requirement is real-time, a first feature fusion model performs weighted fusion based on the signal quality score corresponding to the target decoding signal and the fusion feature to obtain a corresponding fusion feature, wherein the first feature fusion model is a lightweight feature fusion model;

[0039] When the decoding requirement is accuracy, the second feature fusion model performs weighted fusion based on the signal quality score and fusion features corresponding to the target decoding signal to obtain the corresponding fusion features. The second feature fusion model is a deep learning model.

[0040] The decoding requirement is a configuration item, which can be pre-configured or dynamically adjusted during the intent recognition process. In this implementation, a suitable feature fusion model is automatically selected based on the configured decoding requirement to further optimize the decoding effect.

[0041] As an implementable method, a method for calculating a signal quality score corresponding to the first physiological signal is:

[0042] A corresponding signal quality score is calculated based on the signal-to-noise ratio, spectral entropy and power ratio of the first physiological signal.

[0043] As an implementable method, a method for calculating a signal quality score corresponding to each second physiological signal includes:

[0044] Based on the root mean square value, median frequency of the electromyographic spectrum and waveform complexity of the electromyographic signal, the corresponding signal quality score is calculated;

[0045] Based on the signal amplitude change, optical signal intensity and spectrum characteristics corresponding to the functional near-infrared signal, the corresponding signal quality score is calculated;

[0046] Based on the acceleration mean deviation, angular velocity stability and signal sampling rate stability corresponding to the attitude signal, the corresponding signal quality score is calculated.

[0047] The present invention also proposes a motion intention decoding system, comprising:

[0048] An acquisition module, configured to acquire first signal data, wherein the first signal data includes one or more physiological signals corresponding to a current exercise period of the user;

[0049] A prediction system, used to generate a corresponding prediction result based on the first signal data and the prediction model, wherein the prediction result is used to indicate the movement intention corresponding to the user's next action;

[0050] A decoding system, used to generate and output a corresponding decoding result based on the first signal data and the intention recognition model, wherein the decoding result is used to indicate the movement intention corresponding to the user's current action;

[0051] The feedback system is used to calculate the degree of deviation between the decoding result and the prediction result, and is also used to reversely update the intention recognition model based on the degree of deviation when the degree of deviation exceeds a preset deviation threshold.

[0052] The present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned methods is implemented.

[0053] The present invention has significant technical effects due to the adoption of the above technical solution:

[0054] While performing real-time decoding based on the first signal data, the present invention also performs a forward-looking prediction of the user's next action. When the deviation between the decoding result of the real-time action and the predicted result of the next action is large, the deviation is used as a feedback signal to optimize the intention recognition model, which can improve the decoding accuracy when the next action is decoded in real time. This design can effectively ensure the accuracy of motion intention decoding in scenarios of complex collaborative tasks or multi-stage tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 It is a schematic diagram of the working process of a motion intention decoding method shown in Embodiment 1 and Embodiment 2 of the present invention;

[0057] Figure 2 is a schematic diagram of the workflow of a motion intention decoding method shown in Embodiment 3 of the present invention;

[0058] Figure 3 It is a schematic diagram of the workflow of a motion intention decoding method shown in Example 4 of the present invention. DETAILED DESCRIPTION

[0059] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numerals in different drawings represent the same or similar elements.

[0060] The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Instead, they are only examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the attached claims.

[0061] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0062] Embodiment 1: A method for decoding movement intention, based on a single physiological signal, performs movement intention decoding, referring to Figure 1 , including the following steps:

[0063] A100, obtaining first signal data;

[0064] The first signal data is a physiological signal corresponding to the user's current exercise period;

[0065] Those skilled in the art may select a suitable physiological signal as the first signal data according to actual needs. In this embodiment, the first signal data is an electroencephalogram signal.

[0066] A200, generating a corresponding prediction result based on the first signal data and the prediction model;

[0067] The prediction result is used to indicate the movement intention corresponding to the user's next action;

[0068] In this embodiment, the input of the prediction model is the EEG signal, and the output is the prediction result corresponding to the next action.

[0069] In determining the learning goal of the first signal data and predicting the motion intention corresponding to the next action, those skilled in the art can construct the corresponding prediction model based on the existing technology, and this specification does not limit it in detail.

[0070] A300, based on the first signal data and the intention recognition model, generating and outputting a corresponding decoding result;

[0071] The decoding result is used to indicate the movement intention corresponding to the user's current action;

[0072] In this embodiment, the intention recognition model is a unimodal decoding model, the input of which is the corresponding feature data, and the output is the decoding result corresponding to the current action. The feature data is the decoding feature corresponding to the first signal data, namely the EEG signal.

[0073] Specifically:

[0074] Extract features of the EEG signal to obtain corresponding decoding features. In this embodiment, the decoding features include frequency domain features (such as α wave and β wave energy), spectral entropy (SE) and time domain features (such as P300 waveform) corresponding to the EEG signal.

[0075] The decoding features are input into a unimodal decoding model, and the unimodal decoding model outputs a corresponding decoding result.

[0076] As above, in determining the first signal data and the learning goal of identifying the motion intention of the current action, a person skilled in the art may obtain a corresponding unimodal decoding model based on prior art training, and this specification does not limit it in detail.

[0077] A400. Calculate the degree of deviation between the decoding result and the prediction result.

[0078] In this embodiment, the decoding result is a vector, and the prediction result is a scalar. When calculating the degree of deviation, the calculation can be performed directly based on the magnitude of the decoding result and the prediction result. For example, those skilled in the art can determine the degree of deviation by using Mean Absolute Error (MAE);

[0079] In this embodiment, based on a step size of step and a sliding window of n, n pairs of decoding results and prediction results are obtained to calculate the degree of deviation. Those skilled in the art can set the values ​​of step and n according to actual needs. Specifically, the average value of the absolute errors of n consecutive groups of decoding results and prediction results (including the current decoding results and prediction results) is calculated as the degree of deviation. The calculation formula is as follows.

[0080]

[0081] in:

[0082] Prediction To predict the results, Decoding The value of the decoding result.

[0083] A500. When the deviation exceeds a preset deviation threshold, the intention recognition model is reversely updated based on the deviation.

[0084] When the deviation exceeds the preset deviation threshold, it means that the difference between the current action and the next action is large. If the current intention recognition model is continued to be used in the next step, the decoding effect is not good. In this embodiment, the deviation is used as a feedback signal, and the model parameters of the intention recognition model are optimized based on the deviation. When identifying the next action, the motion intention decoding is performed based on the updated intention recognition model, so that the intention recognition model can perform more accurate intent decoding according to the user's individual characteristics and dynamic changes.

[0085] Embodiment 2: A method for decoding movement intention based on multimodal physiological signal fusion decoding;

[0086] In this embodiment:

[0087] The first signal data includes several physiological signals;

[0088] The input of the prediction model is part or all of the physiological signals in the first signal data, and the output is the prediction result corresponding to the next action;

[0089] The intention recognition model is a multimodal decoding model, the input of which is the corresponding fusion feature, and the output of which is the decoding result corresponding to the current action.

[0090] In this embodiment, the features corresponding to each physiological signal are fused based on the feature fusion model to obtain the fused feature, that is, the input of the feature fusion model is the features corresponding to each physiological signal in the first signal data, and the output is the corresponding fused feature.

[0091] Reference Figure 1 , as an example embodiment;

[0092] B100, obtaining first signal data;

[0093] In this embodiment, the first signal data includes an electroencephalogram signal, an electromyography signal, a functional near infrared signal and a posture signal;

[0094] Those skilled in the art may select a variety of physiological signals as the first signal data according to actual needs.

[0095] B200, generating corresponding prediction results based on the first signal data and the prediction model;

[0096] B210, using part or all of the physiological signal in the first signal data as the second signal data;

[0097] In this embodiment, the EEG signal, the EMG signal and the posture signal are used as the second signal data;

[0098] Those skilled in the art may determine the number and types of physiological signals in the second signal data based on the first signal data according to actual needs.

[0099] B220, pre-build prediction models;

[0100] Those skilled in the art may construct a corresponding prediction model on their own according to actual needs, based on the first signal data and the learning goal of predicting the motion intention corresponding to the next action.

[0101] In this embodiment, the prediction model is a pre-built digital twin prediction model, and the method for building the digital twin prediction model includes:

[0102] Obtain sample training data, which can collect the user's EEG signals, EMG signals and posture signals during exercise in advance to obtain corresponding sample training data;

[0103] Use reinforcement learning (RL) algorithms, such as model predictive control (MPC) algorithms;

[0104] The twin model is trained using the sample training data to obtain a digital twin prediction model that can simulate the individual intention pattern of the target user. In real-time tasks, the digital twin prediction model is used to predict the target user's next possible intention or action.

[0105] B230. Input the second signal data into a pre-built digital twin prediction model, and the digital twin prediction model outputs a corresponding prediction result.

[0106] B300, based on the first signal data and the intention recognition model, generating and outputting a corresponding decoding result;

[0107] Specifically:

[0108] B310, extracting features of each physiological signal respectively;

[0109] In this embodiment:

[0110] Extract the frequency domain features (such as alpha wave and beta wave energy), spectral entropy (SE) and time domain features (such as P300 waveform) corresponding to the EEG signal;

[0111] Extract the root mean square (RMS) and spectrum power density of the electromyographic signal.

[0112] Extract the features such as the changes of oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR) corresponding to the functional near-infrared signal;

[0113] The features corresponding to the posture signal, such as Mean Deviation of Acceleration (MDA), Rate of Acceleration Change (RAC), Cumulative Rotation Angle (CRA), Tilt Angle (TA) and Step Frequency (SF), are extracted.

[0114] B320, feature fusion;

[0115] Based on the feature fusion model, the features of each physiological signal are fused to obtain the corresponding fusion features.

[0116] Those skilled in the art may select an existing disclosed feature fusion model according to actual needs.

[0117] B330, motion intention decoding:

[0118] The obtained fusion features are input into the multimodal decoding model, and the multimodal decoding model outputs the corresponding decoding results.

[0119] When the input of the model is determined to be a fusion feature and the learning goal is to decode the motion intention of the current action, those skilled in the art can obtain the corresponding multimodal decoding model based on prior art training, and this specification does not limit it in detail.

[0120] B400. Calculate the degree of deviation between the decoding result and the prediction result.

[0121] B500. When the deviation exceeds a preset deviation threshold, reversely update the multimodal decoding model based on the deviation.

[0122] Embodiment 3: A motion intention decoding method, referring to Figure 2 , the step of “generating a corresponding decoding result based on the first signal data and the intention recognition model” in Example 2 is further optimized, and the rest is the same as Example 2;

[0123] In this embodiment, the EEG signal is used as the first biological signal, and the myoelectric signal, the functional near infrared signal and the posture signal are used as the second biological signal;

[0124] In this embodiment, the intention recognition model includes a unimodal decoding model and a multimodal decoding model;

[0125] The optimized decoding steps are as follows:

[0126] C100, comparing the signal quality score corresponding to the first physiological signal with a preset first score threshold;

[0127] Those skilled in the art can set the quality scoring rule corresponding to the first physiological signal (ie, EEG signal) according to actual needs;

[0128] In this embodiment, the signal quality score corresponding to the EEG signal is calculated based on the signal-to-noise ratio, spectrum entropy and power ratio corresponding to the EEG signal, and the calculation formula is:

[0129] Q EEG =( EE W1 × SNR norm + EE W2 × SE norm + EE W3 × ABR norm ) × 100;

[0130] in:

[0131] Q EEG Indicates the signal quality score corresponding to the EEG signal;

[0132] EE W1 , EE W2 , EE W3 is the feature weight corresponding to each decoded feature of the EEG signal, and those skilled in the art can set it according to actual needs, so that EE W1 + EE W2 + EE W3 =1, in this embodiment EE W1 =0.4, EE W2 =0.3 、eeg W3 =0.3;

[0133] SNR norm , SE norm , ABR norm is the eigenvalue of the corresponding feature after normalization, ranging from [0 1];

[0134] SNR norm represents the corresponding signal-to-noise ratio eigenvalue;

[0135] SE norm represents the corresponding spectral entropy eigenvalue;

[0136] ABR norm represents the corresponding power ratio eigenvalue.

[0137] The following is a detailed introduction to the acquisition methods of EEG signal characteristics such as signal-to-noise ratio, spectral entropy and power ratio:

[0138] Signal-to-Noise Ratio (SNR): It is used to indicate the energy ratio of the effective component of the corresponding signal to the noise, which can reflect the clarity of the signal quality. The calculation formula is:

[0139] .

[0140] Spectral Entropy (SE): It is used to indicate the distribution complexity of the corresponding spectrum. When its value is low, it means that the signal is more concentrated. When it is high, it means that there may be more noise interference. The calculation formula is:

[0141] ;

[0142] in, p(f) is the normalized power spectrum density, which is used to represent the power distribution of the signal at different frequencies.

[0143] Power ratio (Alpha / Beta Ratio, ABR): in EEG α Waves (8-13 Hz) and β wave (13-30 Hz) energy ratio;

[0144] A high power value indicates a state of relaxation, while a low value reflects artifact interference or a task state.

[0145] The calculation formula is as follows:

[0146] ;

[0147] in:

[0148] α power express α wave energy;

[0149] β power express β Wave energy.

[0150] C200, when the signal quality score corresponding to the first physiological signal Q EEG Reaching the first scoring threshold Q Thr1 When the first physiological signal is decoded in a single mode, a corresponding decoding result is obtained;

[0151] That is, when Q EEG ≥ Q Thr1 , it indicates that the EEG signal quality is high and can be used alone for intent decoding.

[0152] Those skilled in the art can set the first scoring threshold according to actual needs. Q Thr1 , this embodiment does not limit it in detail.

[0153] In this embodiment, the first physiological signal, namely the EEG signal, is decoded by a single-modality decoding model to obtain a corresponding decoding result, which is specifically:

[0154] C210, extracting a decoding feature corresponding to the first physiological signal;

[0155] In this embodiment, the decoding features corresponding to the first physiological signal include frequency domain features (such as alpha wave and beta wave energy), spectral entropy (SE) and time domain features (such as P300 waveform);

[0156] C220: Input the obtained decoding features into a unimodal decoding model, and the unimodal decoding model outputs a corresponding decoding result.

[0157] C300, when the signal quality score corresponding to the first physiological signal Q EEG The first scoring threshold is not reached Q Thr1 When the first physiological signal is used as the target decoding signal, and the second physiological signal whose signal quality score is higher than the second score threshold is used as the target decoding signal, the multimodal fusion decoding is performed on each target decoding signal;

[0158] That is, when Q EEG < Q Thr1 When , it indicates that the signal quality of the EEG signal is poor and needs to be combined with other signals for multimodal fusion decoding.

[0159] The specific steps are:

[0160] C310, taking a second physiological signal whose signal quality score reaches a preset second score threshold as a target decoding signal;

[0161] Those skilled in the art can set the quality scoring rules corresponding to the second physiological signal (i.e., electromyographic signal, functional near infrared signal and posture signal) according to actual needs;

[0162] For electromyography (EMG):

[0163] In this embodiment, the signal quality score corresponding to the electromyographic signal EMG is calculated based on the root mean square value corresponding to the electromyographic signal EMG, the median frequency of the electromyographic spectrum, and the waveform complexity. The calculation formula is:

[0164] Q EMG =( emg W1 × RMS norm + emg W2 × MPF norm + emg W3 × ZCR norm ) × 100;

[0165] in:

[0166] Q EMG Indicates the signal quality score corresponding to the electromyographic signal EMG;

[0167] emg W1 , emg W2 , emg W3 is the feature weight corresponding to each decoding feature of the electromyographic signal EMG, and those skilled in the art can set it according to actual needs, so that emg W1 + emg W2 + emg W3 =1, in this embodiment emg W1 =0.4, emg W2 =0.4, emg W3 =0.2;

[0168] RMS norm , MPF norm , ZCR norm is the eigenvalue of the corresponding feature after normalization, ranging from [0 1];

[0169] RMS norm represents the corresponding RMS eigenvalue;

[0170] MPF norm Indicates the corresponding median frequency characteristic value of the electromyographic spectrum;

[0171] ZCR norm Indicates the corresponding waveform complexity eigenvalue.

[0172] The following is a detailed introduction to the methods of obtaining the RMS value of the electromyographic signal, the median frequency of the electromyographic spectrum, and the complexity of the waveform:

[0173] Root Mean Square (RMS): reflects the energy characteristics of the electromyographic signal. When the noise ratio is high, the RMS value is low. The calculation formula is as follows:

[0174]

[0175] in, N represents the total number of sampling points, x i For the i The signal sampling value corresponding to the sampling point.

[0176] Median Frequency (MDF) of the EMG spectrum: represents the frequency distribution characteristics of the EMG power spectrum. A lower median frequency of the EMG spectrum indicates artifacts or fatigue. In this embodiment, the power spectrum density corresponding to the EMG signal is calculated, and the frequency when the power of the frequency range is integrated to half of the total power is taken as the median frequency MDF of the EMG spectrum.

[0177] Waveform complexity (Zero Crossing Rate, ZCR): used to reflect the change pattern of the electromyographic signal. In this embodiment, the waveform complexity is determined based on the signal zero crossing rate (ZCR) per unit time.

[0178] Targeting functional near-infrared signals fNIRS:

[0179] In this embodiment, the signal quality score corresponding to the functional near infrared signal is calculated based on the signal amplitude change, light signal intensity and spectrum characteristics corresponding to the functional near infrared signal. The calculation formula is:

[0180] Q fNIRS =( fnirs W1 × AV norm + fnirs W2 × OSI norm + fnirs W3 × FF norm ) × 100;

[0181] in:

[0182] Q fNIRS It represents the signal quality score corresponding to the functional near-infrared signal fNIRS;

[0183] fnirs W1 , fnirs W2 , fnirs W3 is the feature weight corresponding to each decoding feature of the functional near infrared signal fNIRS, and those skilled in the art can set it according to actual needs. fnirs W1 + fnirs W2 + fnirs W3 =1, in this embodiment fnirs W1 =0.4, fnirsW2 =0.4, fnirs W3 =0.2;

[0184] AV norm , OSI norm , FF norm is the eigenvalue of the corresponding feature after normalization, ranging from [0 1];

[0185] AV norm Indicates the corresponding signal amplitude change characteristic value;

[0186] OSI norm Indicates the corresponding optical signal intensity characteristic value;

[0187] FF norm Indicates the corresponding spectral eigenvalue.

[0188] The following is a detailed introduction to the method of obtaining the characteristics of the signal amplitude change, optical signal intensity and spectrum power corresponding to the functional near-infrared signal:

[0189] Signal Amplitude Variability (AV): used to indicate the amplitude of oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR) in the corresponding signal changing over time. In this embodiment, it is obtained by calculating the difference between the peak value and the valley value of the corresponding signal.

[0190] Optical Signal Intensity (OSI): In this embodiment, the detector collects the corresponding light intensity and calculates the average value, and the obtained average value is used as the optical signal intensity. A low optical signal intensity may indicate a high signal-to-noise ratio.

[0191] The calculation formula is as follows:

[0192]

[0193] in:

[0194] T represents the total number of corresponding sampling points

[0195] I(t) is the light intensity corresponding to the tth sampling point.

[0196] Frequency Features (FF): used to analyze whether the low-frequency components (such as 0.01-0.2 Hz) of the HbO and HbR signals conform to the typical neurovascular coupling pattern. In this embodiment, the spectral power of the corresponding signal is extracted based on the fast Fourier transform (FFT) to obtain the corresponding spectral features.

[0197] For attitude signal MPS:

[0198] In this embodiment, the signal quality score corresponding to the posture signal MPS is calculated based on the acceleration mean deviation, angular velocity stability and signal sampling rate stability corresponding to the posture signal MPS. The calculation formula is:

[0199] Q MPS =( mps W1 × MDA norm + mps W2 × AVS norm + mps W3 × SRS norm ) × 100;

[0200] in:

[0201] Q MPS Indicates the signal quality score corresponding to the posture signal MPS;

[0202] mps W1 , mps W2 , mps W3 is the feature weight corresponding to each decoded feature of the posture signal MPS, and those skilled in the art can set it according to actual needs, so that mps W1 + mps W2 + mps W3 =1, in this embodiment mps W1 =0.4, mps W2 =0.4, mps W3 =0.2;

[0203] MDA norm , AVS norm , SRSnorm is the eigenvalue of the corresponding feature after normalization, ranging from [0 1];

[0204] MDA norm represents the corresponding acceleration mean deviation eigenvalue;

[0205] AVS norm represents the corresponding angular velocity stability eigenvalue;

[0206] SRS norm Represents the corresponding signal sampling rate stability eigenvalue.

[0207] The following is a detailed introduction to the acquisition methods of the acceleration mean deviation, angular velocity stability, and signal sampling rate stability corresponding to the attitude signal:

[0208] Mean Deviation of Acceleration (MDA): That is, the average deviation of the three-axis acceleration from its reference value, calculated as follows:

[0209]

[0210] in:

[0211] M represents the total number of corresponding sampling points;

[0212] a i Indicates i The three-axis acceleration corresponding to the sampling points;

[0213] a 0 Indicates the preset three-axis acceleration reference value;

[0214] Angular Velocity Stability (AVS): That is, the rate of change of the angular velocity of the gyroscope, calculated as follows:

[0215]

[0216] in:

[0217] L represents the total number of corresponding sampling points;

[0218] △w i Indicates i The angular velocity corresponding to the sampling points;

[0219] △t Indicates unit time;

[0220] Signal sampling rate stability (Sampling Rate Stability, SRS): used to indicate whether the sampling interval of the corresponding signal is uniform. In this embodiment, it is calculated by detecting the interval change between consecutive time points.

[0221] As another possible implementation, when the signal quality scores corresponding to all second physiological signals do not reach a preset second score threshold, the decoding mode is switched to single physiological signal decoding, that is, the corresponding EEG signal is decoded based on a single modal decoding model.

[0222] As an implementable method, when the signal quality scores corresponding to all second physiological signals do not reach the preset second score threshold, poor signal quality is fed back to the user, and the user is shown the correct wearing method of the relevant device.

[0223] Note:

[0224] Those skilled in the art may set the node for calculating the signal quality score of each second physiological signal according to actual needs, and may perform a quality score on each second physiological signal after determining that the signal quality score corresponding to the first physiological signal does not reach a preset first score threshold, or may complete the scoring of the second physiological signal after obtaining the second physiological signal, which is not limited in detail in this embodiment;

[0225] The technicians in this field can set the second scoring threshold according to actual needs. Q Thr2 , this embodiment does not limit it in detail.

[0226] C320. Based on the obtained target decoding signal, perform intent decoding through the multimodal decoding model to obtain a corresponding decoding result.

[0227] The specific steps include:

[0228] C321, extracting decoding features corresponding to each target decoding signal;

[0229] C322, performing feature weighting processing based on the signal quality score and decoding features corresponding to each target decoding signal to obtain corresponding fusion features;

[0230] In this embodiment, the feature fusion model performs weighted processing of multimodal signal features based on the signal quality scores and fusion features corresponding to each target decoded signal, and outputs the corresponding feature matrix as the fusion feature. The corresponding weights in the feature fusion process will be dynamically adjusted according to the signal quality scores.

[0231] Those skilled in the art may select a deep learning model, a random forest model or a mean weighted model as a feature fusion model according to actual needs, and this embodiment does not limit it in detail.

[0232] C323. Input the fusion feature into a multimodal decoding model, and the multimodal decoding model outputs a corresponding decoding result.

[0233] In this embodiment:

[0234] The frequency domain features (such as alpha wave and beta wave energy), spectral entropy (SE) and time domain features (such as P300 waveform) corresponding to the EEG signal are used as its decoding features;

[0235] The root mean square (RMS) and spectrum power density of the electromyographic signal are used as decoding features.

[0236] The change characteristics of oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR) corresponding to the functional near-infrared signal are used as the decoding features;

[0237] The features corresponding to the posture signal, such as Mean Deviation of Acceleration (MDA), Rate of Acceleration Change (RAC), Cumulative Rotation Angle (CRA), Tilt Angle (TA) and Step Frequency (SF), are used as decoding features.

[0238] In this embodiment, when the deviation between the corresponding decoding result and the prediction result exceeds a preset deviation threshold, the single-modal decoding model and the multi-modal decoding model are reversely updated based on the deviation to improve the decoding accuracy during the next motion intention decoding.

[0239] Today’s motion intent decoding technology mainly relies on the following two methods:

[0240] Based on single physiological signal decoding:

[0241] This method directly decodes movement intention through signals such as electroencephalogram (EEG), electromyography (EMG), functional near-infrared spectroscopy (fNIRS) or motion and posture signals (MPS).

[0242] This method is simple to implement, but when the signal quality is low, the noise is high, or the individual differences are large, the decoding performance often decreases significantly, affecting the accuracy of intent recognition.

[0243] Based on multimodal signal fusion decoding:

[0244] That is, multimodal signals such as EEG, EMG, and MPS are combined to comprehensively decode movement intention;

[0245] This method improves the decoding accuracy and robustness to a certain extent, but its real-time performance and practicality are limited due to the complexity of the data fusion algorithm and the high requirements on computing resources.

[0246] In summary, the single physiological signal decoding method is highly dependent on signal quality and human functional status. When the signal noise is high or the motor function is severely impaired, its decoding performance decreases significantly and its scope of application is relatively limited. Although multimodal signal fusion decoding improves robustness, it lacks real-time and convenience, and data processing delays in complex scenarios may affect the experience.

[0247] In response to the above-mentioned defects, this embodiment intelligently selects an appropriate signal decoding method according to the characteristics (decoding features) and quality (signal quality score) of various physiological signals, which can not only improve the accuracy and robustness of decoding, but also effectively improve the real-time and adaptability of the system.

[0248] Embodiment 4: A motion intention decoding method, referring to Figure 3 , the step of "performing feature fusion based on the signal quality score and decoding features corresponding to each target decoded signal to obtain corresponding fusion features" in Example 3 is further optimized, and the rest is the same as Example 3;

[0249] In this embodiment, the feature fusion model includes a first feature fusion model and a second feature fusion model;

[0250] The inputs of the first feature fusion model and the second feature fusion model are both the signal quality scores and fusion features corresponding to each target decoded signal, and the outputs are both the corresponding fusion features;

[0251] The first feature fusion model is an existing publicly available lightweight feature fusion model such as a weighted average model or a random forest model;

[0252] The second feature fusion model is an existing publicly available deep learning model such as a Transformer network based on an attention mechanism.

[0253] The optimized fusion steps are as follows:

[0254] D100, obtain the decoding requirements corresponding to the current decoding task;

[0255] Decoding requirements include real-time and accuracy; users can set and adjust decoding requirements according to actual conditions.

[0256] D200. When the decoding requirement is real-time, the first feature fusion model performs weighted fusion based on the signal quality score and fusion features corresponding to the target decoding signal to obtain corresponding fusion features.

[0257] D300. When the decoding requirement is accuracy, the second feature fusion model performs weighted fusion based on the signal quality score and fusion features corresponding to the target decoding signal to obtain the corresponding fusion features.

[0258] This embodiment, through the design of decoding requirements, can dynamically switch the fusion strategy according to the complexity of the task and the real-time requirements during the decoding process;

[0259] For example, for high-precision tasks (such as decoding fine hand movements), a deep learning fusion model is used; while for tasks with high real-time requirements (such as gait recognition), lightweight models such as weighted average or random forest are used.

[0260] In summary, the motion intention recognition method provided in this embodiment comprehensively selects the decoding algorithm and fusion strategy through signal quality, feature distribution and real-time requirements; among them, the signal quality index reflects the noise level and reliability of the signal, the feature distribution index reflects the expression pattern of the motion intention in the signal, and the real-time requirement ensures that the decoding system can meet the dynamic adaptation of complex task scenarios; based on the comprehensive evaluation and algorithm selection of these key indicators, it is possible to achieve accurate recognition of user intentions while ensuring the robustness of decoding and the efficiency of the system, further ensuring the stable performance and excellent effect of the motion intention recognition technology in a variety of application scenarios.

[0261] In this embodiment, the corresponding first feature fusion model, second feature fusion model, single-modal decoding model and multi-modal decoding model are pre-trained. In practical applications, parameters of the single-modal decoding model and the multi-modal decoding model are optimized based on the deviation between the decoding result and the prediction result corresponding to each decoding task.

[0262] In order to improve the universality and robustness of the motion intention recognition method, a signal database and a decoding model library are pre-built in this embodiment:

[0263] Signal database:

[0264] A standardized multimodal signal database is constructed by collecting EEG, EMG, fNIRS and MPS signals from a large number of users of different ages, genders, health conditions and task scenarios. The signal database provides signal quality evaluation indicators and features, providing a basis for signal quality grading and signal weighting strategies in real-time tasks.

[0265] Decoding model library:

[0266] Based on the data samples provided by the signal database, deep learning, random forest and other algorithms are used to train single-modal decoding models, as well as a model combination of the first feature fusion model, the second feature fusion model and the multi-modal decoding model, and to cover task requirements from motion recognition to cognitive state analysis.

[0267] Through the grading of signal quality scores and the design of decoding requirements, this method has an adaptive selection mechanism, which can dynamically select the optimal model combination according to the real-time signal quality and decoding requirements to ensure decoding efficiency and robustness in different scenarios.

[0268] The following is an example of the application of the motion intention recognition method proposed in this embodiment.

[0269] In rehabilitation training, the user wears a brain-computer interface device, and the method identifies the user's movement intention based on the collected physiological signals, and controls the corresponding rehabilitation robot to complete the limb training task based on the movement intention;

[0270] Users and rehabilitation doctors can adjust the intensity and mode of training tasks by observing the synchronization between decoding results and actual movements; this real-time feedback mechanism effectively improves the accuracy and efficiency of rehabilitation training, while enhancing patients' understanding and participation in the training process.

[0271] At the same time, the signal quality score corresponding to each signal can be displayed in real time. In the scenario of remote guidance, the rehabilitation doctor can adjust the training plan based on the signal quality score of each signal, the real-time decoding results and the digital twin prediction results, that is, feedback to the user in real time to adjust the training plan, or adjust the relevant parameters in time (such as the model parameters of the digital twin prediction model, the feature weights corresponding to each physiological signal in the quality assessment, decoding requirements, etc.). For example, when a certain modality signal (such as fNIRS) of the user is detected to be abnormal, the doctor can promptly suggest the user to optimize the wearing method or adjust the training environment.

[0272] Embodiment 5: A system for identifying movement intention, comprising:

[0273] An acquisition module, configured to acquire first signal data, wherein the first signal data includes one or more physiological signals corresponding to a current exercise period of the user;

[0274] A prediction system, used to generate a corresponding prediction result based on the first signal data and the prediction model, wherein the prediction result is used to indicate the movement intention corresponding to the user's next action;

[0275] A decoding system, used to generate and output a corresponding decoding result based on the first signal data and the intention recognition model, wherein the decoding result is used to indicate the movement intention corresponding to the user's current action;

[0276] The intention recognition model is a unimodal decoding model or a multimodal decoding model;

[0277] The feedback system is used to calculate the degree of deviation between the decoding result and the prediction result, and is also used to reversely update the intention recognition model based on the degree of deviation when the degree of deviation exceeds a preset deviation threshold.

[0278] Embodiment 6, a motion intention recognition system, further optimizes the decoding system on the basis of embodiment 5, and the rest is the same as embodiment 5;

[0279] In this embodiment, the first signal data includes a first physiological signal and a plurality of second physiological signals, wherein the first physiological signal is an electroencephalogram signal; and the second physiological signal is an electromyography signal, a functional near infrared signal, and a posture signal.

[0280] The decoding system includes:

[0281] A quality assessment model, used to calculate the signal quality score corresponding to each physiological signal;

[0282] a judgment module, used to compare the signal quality score of the first physiological signal with a preset first scoring threshold, and also used to compare the signal quality score of each second physiological signal with a preset second scoring threshold;

[0283] a feature extraction module, configured to extract decoding features of a target decoding signal, and when a signal quality score of a first physiological signal reaches a preset first scoring threshold, the first physiological signal is used as a target decoding signal; otherwise, the first physiological signal and a second physiological signal whose signal quality score reaches a preset second scoring threshold are both used as target decoding signals;

[0284] A single-modality decoding module, when there is only one target decoding signal, inputs the decoding feature corresponding to the first physiological signal into a single-modality decoding model, and the single-modality decoding model outputs a corresponding decoding result;

[0285] A fusion module, when there are multiple target decoding signals, performs feature fusion based on the signal quality scores and decoding features corresponding to each target decoding signal to obtain a fusion feature;

[0286] The multimodal decoding module inputs the fusion features into the single-modal decoding model, and the multimodal decoding model outputs the corresponding decoding results.

[0287] Embodiment 7, a motion intention recognition system, further optimizes the fusion module on the basis of embodiment 6, and the rest is the same as embodiment 6;

[0288] The fusion module includes:

[0289] A first feature fusion model is used to perform weighted fusion based on the signal quality score corresponding to the target decoded signal and the fusion feature to obtain a corresponding fusion feature, wherein the first feature fusion model is a lightweight feature fusion model;

[0290] A second feature fusion model is used to perform weighted fusion based on the signal quality score corresponding to the target decoded signal and the fusion feature to obtain a corresponding fusion feature, wherein the first feature fusion model is a deep learning model;

[0291] The priority judgment unit is used to obtain the decoding requirements corresponding to the current decoding task, and determine whether the first feature fusion model or the second feature fusion model is used to perform feature fusion work according to the decoding requirements.

[0292] As for the above-mentioned device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0293] Embodiment 8: A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the above embodiments 1 to 4 is implemented.

[0294] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0295] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0296] The present invention is described with reference to the flowcharts and / or block diagrams of the method, terminal device (system), and computer program product according to the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0297] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0298] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0299] It should be noted that:

[0300] The "one embodiment" or "embodiment" mentioned in the specification means that the specific features, structures or characteristics described in conjunction with the embodiment are included in at least one embodiment of the present invention. Therefore, the phrases "one embodiment" or "embodiment" appearing in various places throughout the specification do not necessarily refer to the same embodiment.

[0301] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0302] In addition, it should be noted that the shapes and names of the parts and components of the specific embodiments described in this specification may be different. Any equivalent or simple changes made based on the structure, features and principles described in the patent concept of the present invention are included in the protection scope of the patent of the present invention. The technicians in the technical field of the present invention can make various modifications or supplements to the specific embodiments described or replace them in a similar manner, as long as they do not deviate from the structure of the present invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. A motion intention decoding method, characterized in that: The following steps are involved: Acquire first signal data, where the first signal data includes one or more physiological signals corresponding to the current exercise period of the user, wherein the first signal data includes a first physiological signal and a plurality of second physiological signals, and the first physiological signal is an electroencephalogram signal; Based on the first signal data and the prediction model, a corresponding prediction result is generated, where the prediction result is used to indicate the movement intention corresponding to the user's next action; Based on the first signal data and the intention recognition model, a corresponding decoding result is generated and output, wherein the decoding result is used to indicate the motion intention corresponding to the user's current action, wherein the intention recognition model includes a unimodal decoding model and a multimodal decoding model; Calculating the degree of deviation between the decoding result and the prediction result; When the deviation exceeds a preset deviation threshold, the intention recognition model is reversely updated based on the deviation. Specifically, when the deviation between the corresponding decoding result and the prediction result exceeds a preset deviation threshold, the single-mode decoding model and the multi-modal decoding model are reversely updated based on the deviation. The specific steps of generating a corresponding decoding result based on the first signal data and the intention recognition model include: Comparing the signal quality score corresponding to the first physiological signal with a preset first score threshold; When the signal quality score corresponding to the first physiological signal reaches the first score threshold, decoding the first physiological signal based on the single-modality decoding model to obtain a corresponding decoding result; When the signal quality score corresponding to the first physiological signal does not reach the first scoring threshold, the first physiological signal is used as the target decoding signal, and the second physiological signal with a signal quality score higher than the second scoring threshold is used as the target decoding signal. The multimodal decoding model performs multimodal fusion decoding based on each target decoding signal to obtain a corresponding decoding result.

2. The motion intention decoding method according to claim 1, characterized in that: The second physiological signal is an electromyographic signal, a functional near infrared signal and a posture signal.

3. The motion intention decoding method according to claim 2, characterized in that: The prediction model is a digital twin prediction model; The input of the digital twin prediction model is EEG signals, EMG signals and posture signals, and the output is the corresponding prediction results.

4. The motion intention decoding method according to any one of claims 1 to 3, characterized in that: The specific steps of performing multimodal fusion decoding based on each target decoding signal by the multimodal decoding model are as follows: Extracting decoding features corresponding to each target decoding signal; Perform feature weighting processing based on the signal quality score and decoding features corresponding to each target decoding signal to obtain corresponding fusion features; The fusion feature is input into the multimodal decoding model, and the multimodal decoding model outputs a corresponding decoding result.

5. The motion intention decoding method according to claim 4, characterized in that: Based on the signal quality score and decoding features corresponding to each target decoded signal, feature weighting processing is performed to obtain the corresponding fusion features in the following specific steps: Get the decoding requirements corresponding to the current decoding task; When the decoding requirement is real-time, a first feature fusion model performs weighted fusion based on the signal quality score corresponding to the target decoding signal and the fusion feature to obtain a corresponding fusion feature, wherein the first feature fusion model is a lightweight feature fusion model; When the decoding requirement is accuracy, the second feature fusion model performs weighted fusion based on the signal quality score and fusion features corresponding to the target decoding signal to obtain the corresponding fusion features. The second feature fusion model is a deep learning model.

6. The motion intention decoding method according to claim 1, characterized in that: The method for calculating the signal quality score corresponding to the first physiological signal is: A corresponding signal quality score is calculated based on the signal-to-noise ratio, spectral entropy and power ratio of the first physiological signal.

7. The motion intention decoding method according to claim 2, characterized in that: The calculation method of the signal quality score corresponding to each second physiological signal includes: Based on the root mean square value, median frequency of the electromyographic spectrum and waveform complexity of the electromyographic signal, the corresponding signal quality score is calculated; Based on the signal amplitude change, optical signal intensity and spectrum characteristics corresponding to the functional near-infrared signal, the corresponding signal quality score is calculated; Based on the acceleration mean deviation, angular velocity stability and signal sampling rate stability corresponding to the attitude signal, the corresponding signal quality score is calculated.

8. A motion intention decoding system, characterized in that: include: An acquisition module, configured to acquire first signal data, wherein the first signal data includes one or more physiological signals corresponding to the current exercise period of the user, wherein the first signal data includes a first physiological signal and a plurality of second physiological signals, and the first physiological signal is an electroencephalogram signal; A prediction system, used to generate a corresponding prediction result based on the first signal data and the prediction model, wherein the prediction result is used to indicate the movement intention corresponding to the user's next action; A decoding system, configured to generate and output a corresponding decoding result based on the first signal data and the intention recognition model, wherein the decoding result is used to indicate the motion intention corresponding to the user's current action, wherein the intention recognition model includes a unimodal decoding model and a multimodal decoding model; A feedback system, used to calculate the degree of deviation between the decoding result and the prediction result, and also used to reversely update the intention recognition model based on the degree of deviation when the degree of deviation exceeds a preset deviation threshold, specifically, when the degree of deviation between the corresponding decoding result and the prediction result exceeds a preset deviation threshold, reversely update the unimodal decoding model and the multimodal decoding model based on the degree of deviation; The decoding system is configured to: Comparing the signal quality score corresponding to the first physiological signal with a preset first score threshold; When the signal quality score corresponding to the first physiological signal reaches the first score threshold, decoding the first physiological signal based on the single-modality decoding model to obtain a corresponding decoding result; When the signal quality score corresponding to the first physiological signal does not reach the first scoring threshold, the first physiological signal is used as the target decoding signal, and the second physiological signal with a signal quality score higher than the second scoring threshold is used as the target decoding signal. The multimodal decoding model performs multimodal fusion decoding based on each target decoding signal to obtain a corresponding decoding result.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images