Autonomous action prediction method and device based on sequential spatiotemporal brain and muscle electrical signals

Abstract

The application discloses an autonomous action prediction method based on a sequential space-time domain brain and muscle electric signal, first, a space-time graph network matrix is constructed, space-time data mapping is performed on the collected brain and muscle electric signal of a patient, and the signal is converted into a graph network matrix corresponding to a plurality of continuous time frames in a data window division manner; then, a graph network matrix of a future signal is predicted, the graph matrix data of the plurality of continuous time frames are sent into a graph convolutional neural network for analysis, and space-time data prediction is performed on the graph matrix corresponding to the future time frames; finally, target action recognition is performed, the predicted plurality of time frame graph matrices are subjected to pattern recognition through the graph convolutional neural network containing a plurality of full connection layers, the action corresponding to the signal is analyzed, and the corresponding action of a control auxiliary device is output in advance. The relatively high explainability of the algorithm can help popularize and apply the algorithm in the rehabilitation medical field, thereby helping more stroke patients.

Description

Technical Field

[0001] This invention belongs to the field of physiological signal processing, specifically relating to a method and device for predicting autonomous movements based on sequential spatiotemporal brain electromyography signals. Background Technology

[0002] Physiological signal analysis, as an important research direction in human-computer interfaces, has numerous applications in fields such as computers, electronic devices, medicine, and sports. Among these, the applications of physiological signals such as electroencephalography (EEG), electromyography (EMG), electrodermal conductance (EDC), and electrocardiography (ECG) are the most common. In the field of rehabilitation medicine, by collecting and analyzing the relevant signals from patients and extracting the features contained within them, it is possible to effectively obtain the patient's physiological information, facilitating diagnosis by doctors and promoting the movement of rehabilitation equipment. This approach has been proven effective, especially for stroke patients caused by stroke.

[0003] In terms of specific rehabilitation implementation, current applications can be broadly divided into two main directions: rehabilitation assessment and rehabilitation training. Rehabilitation assessment is mainly used to evaluate and score the patient's rehabilitation status and assist physicians in making diagnoses; while rehabilitation training focuses on analyzing the patient's physiological signals to drive peripheral components to assist the patient in rehabilitation training and stimulate the establishment of neural pathways.

[0004] Because the construction of rehabilitation training methods requires the coordination of peripheral components and a deeper level of human-computer interaction, assessment-oriented assistive methods are currently more common than training-oriented methods. However, from a user perspective, the former often only serves as one of the criteria for physician diagnosis and treatment; while the latter allows patients to complete training independently without the accompaniment of a physician.

[0005] Currently, most rehabilitation training methods based on physiological signals adopt the approach of "prompting action - detecting completion status - external compensation". That is, the system guides the patient to complete a certain action through interface diagrams, voice and other means. During the process of the patient completing the action, the system collects the patient's electroencephalogram (EEG) and electromyogram (EMG) signals to analyze the completion status of the action. If the completion status is not good, the system uses assistive devices (such as robotic arms, electrical stimulation, etc.) to help the patient complete the action.

[0006] However, in this situation, the patient's movements are based on external cues, rather than being entirely asynchronous and driven by their own whims. The lack of fully voluntary motor intention during movement hinders the establishment of neural pathways. Therefore, "allowing patients to generate their own motor intentions before moving" is currently recognized by rehabilitation physicians as a more beneficial training method for patient recovery.

[0007] In recent years, with the gradual development and application of machine learning algorithms, introducing intelligent analysis algorithms into the analysis of physiological signals has gradually become a new development direction in the field of rehabilitation medicine, and some rehabilitation methods based on machine learning algorithms have been proposed. The main logic of these methods is to collect one or more physiological signals, perform pattern recognition on the collected signals, determine the current action of the patient, and then drive assistive devices to compensate for the actions that the patient has not fully performed.

[0008] However, these methods often have the following most critical shortcomings:

[0009] Almost all current methods suffer from a lag in assistive device activation. This is because current physiological signal recognition algorithms, in order to ensure accuracy, often require a certain amount of time after the action has occurred before collecting a sufficient length of signal for pattern recognition. Only after analyzing the action does the assistive device begin to move. This process causes a noticeable sense of frustration for the patient attempting movement. This frustration significantly interferes with the establishment of the patient's "voluntary movement" neural pathways, hindering rehabilitation.

[0010] Especially during the daily rehabilitation movements that awaken patients, various corresponding physiological manifestations (such as muscle contraction, brain activity, eye movements, and cardiac tremors) exhibit a sequential nature. This sequence is represented as a "sequential" signal with a time sequence during the data collection process. For example, when a subject performs a "forceful arm swing," the motor brain region first shows active signaling, followed by sequential firing of the upper and lower arm muscles, and then an increase in cardiac frequency. However, current algorithm models lack the ability to analyze the sequential characteristics of these signals across the time dimension. This analysis method loses many of the inherent features of the signals themselves, making the lag in assistive device response even more pronounced. Summary of the Invention

[0011] The technical problem to be solved by the present invention is to provide a method and device for predicting autonomous movements based on sequential spatiotemporal electromyography signals, which solves the problem in the prior art where the lag in the driving of auxiliary devices interferes with the establishment of neural pathways for rehabilitation.

[0012] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0013] A method for predicting autonomous movements based on sequential spatiotemporal brain electromyography signals.

[0014] First, a spatiotemporal graph network matrix is ​​constructed. Electromyography and electroencephalography signals of stroke patients are collected. By dividing the time window and splicing along the lead dimension, a spatiotemporal graph network matrix corresponding to each window time frame is established.

[0015] Then, the graph network matrix for predicting future signals is obtained by feeding graph matrix data from multiple consecutive time frames into a graph convolutional neural network for analysis. By analyzing the diffusion and aggregation of signals between leads, the graph network matrix corresponding to several future time frames is predicted.

[0016] Finally, for target action recognition, the predicted multiple time frame matrixes are concatenated with some of the real time frame matrixes to obtain an action discrimination input sequence containing long-term temporal features. After inter-lead diffusion through graph convolutional layers, the sequence is fed into a multi-layer fully connected layer for pattern recognition, predicts the action corresponding to the signal, and outputs the corresponding action of the control auxiliary device in advance.

[0017] The specific process of constructing the spatiotemporal graph network matrix is ​​as follows:

[0018] First, collect electroencephalogram (EEG) signals and surface electromyography (EMG) signals of the designated muscles;

[0019] Secondly, the collected signals are divided into multiple time windows with a certain overlap rate according to their sequence;

[0020] Then, the EEG and EMG signals within each time window are spliced ​​and fused to establish a two-dimensional signal matrix, resulting in a sequential two-dimensional EEG and EMG signal map for each data frame.

[0021] The specific process of using a graph network matrix to predict future signals is as follows:

[0022] First, a signal matrix of a specified length of data frame is selected for prediction;

[0023] Then, the data frame is fed into the graph convolutional neural network in a temporal manner to perform spatiotemporal data prediction, and the prediction results of several data frames are obtained.

[0024] The graph convolutional neural network includes two parallel graph structure adjacency matrices for performing graph convolution: a static adjacency matrix of the physiological association between the measured brain regions and muscle blocks, and an adaptive dynamic graph adjacency matrix obtained by multiplying the attention vectors.

[0025] The static adjacency matrix of the physiological association graph between the measured brain regions and muscle blocks is a fixed empirical hyperparameter based on the patient's condition and task classification, and will not change with the increase of iterations in the early training.

[0026] The dynamic graph adjacency matrix obtained by multiplying the adaptive attention vectors is continuously iterated during the model training process. The attention vectors explore the correlation between different lead signals within the physiological features and obtain the internal connections for the sequential signals themselves.

[0027] The specific process of target action recognition is as follows:

[0028] The last MN data frames from the T data frames actually collected are combined with the predicted N data frames to form a data frame sequence of length M, which is used to predict the actions the patient is about to perform. During the prediction process, the action prediction module is used to perform pattern recognition on the sequential signal composed of the entire M data frames and analyze its corresponding action paradigm, where M>N.

[0029] In the process of pattern recognition, the sequential signal graph consisting of M data frames is first passed through a graph convolutional network. Then, the intermediate result after graph convolution is expanded and sent to two consecutive fully connected layers for target action recognition.

[0030] The collected electromyographic signals from the patient's brain consist of two parts. One part consists of electromyographic signals collected during a rehabilitation patient's performance of a certain paradigmatic movement. These signals need to be labeled for training the pattern recognition model. The other part consists of electromyographic signals collected during daily rehabilitation training while the rehabilitation patient wears the acquisition device. These signals do not need to be labeled and are used for training the prediction model. A large number of these signals need to be collected.

[0031] After the pattern recognition model and the prediction model are trained, the two models are concatenated according to the logic of "predict first and then recognize", and trained using real data with labels. During the training process, the parameters of the two models are fine-tuned, and both the predicted sequence and the recognized action are used as the loss function of the overall model.

[0032] The autonomous action prediction device based on sequential spatiotemporal brain-myomyography signals includes a brain-myomyography signal acquisition device, a signal processing device, and auxiliary devices. The brain-myomyography signal acquisition device is used to acquire the patient's brain and muscle signals. The signal processing device is used to train a sequence prediction model and an action recognition model based on the acquired brain and muscle signals, and to issue a predicted action signal. The auxiliary devices are used to receive the predicted action signal and issue a corresponding prompt signal.

[0033] The electromyography (EMG) signal acquisition device and auxiliary devices are all wearable devices, and can be selected and worn on the corresponding parts of the patient according to the severity of the condition.

[0034] A computer storage medium storing computer-readable instructions that, when executed by a processor, invoke all or part of the steps of the method.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1) Compared to the currently common "prompt-completion-external compensation" rehabilitation training method, this method provides a training approach that allows patients to perform any type of movement at any time. Under this rehabilitation training method, all the patient's movements are generated and performed spontaneously by the patient, without relying on any form of external prompting or requirement, possessing complete autonomy and more effectively assisting the patient in establishing neural pathways.

[0037] 2) This method does not perform pattern recognition of movements based on real-time EMG signals. Instead, it predicts movements in several future time frames based on current EMG signals, thereby activating the assistive device in advance to achieve complete synchronization with the patient in performing the predicted movement pattern. The patient will not experience any lag during use and will perceive the assistive device assisting them in performing the desired movement in perfect synchronization. This method is undoubtedly the most effective assistance for patients, allowing them to fully autonomously decide when to perform which movement while receiving complete synchronization assistance from the device.

[0038] 3) This method models and analyzes the sequential features in brain electromyography (EMG) signals by dividing data frames and establishing a spatiotemporal matrix. It analyzes the sequential relationships between different brain regions and muscle blocks, extracting more temporal information compared to other algorithms. Based on the analysis of sequential signal features, this method uses hybrid graph convolution analysis of the spatiotemporal matrix to predict future data frames and anticipate upcoming patient actions.

[0039] 4) Compared to general GCN models, this method proposes to construct adjacency matrices based on surface electrode distances and self-learning graph adjacency matrices based on adaptive attention in parallel to address the correlations between multiple brain regions and muscle blocks. During training and recognition, the results of convolution of the two graphs are fused at the gate level. This allows the GCN model to generate an attention mechanism for the spatiotemporal matrix of signals without losing generality due to distance relevance, enabling better analysis of the connections between multiple signals. Simultaneously, it provides a research basis for exploring the correlation information of signal leads beyond the anatomical background.

[0040] 5) This method employs a GCN model with an adaptive graph structure for algorithm design. Compared to other algorithms, graph convolution-based methods offer more interpretability and analytical possibilities. The relatively high interpretability of this algorithm facilitates its widespread application in the field of rehabilitation medicine, thereby helping more stroke patients. Simultaneously, the self-learning adjacency graph matrix generated based on the self-attention mechanism provides a new perspective for exploring the connections between different physiological representations, helping people better understand the mechanism by which sequential signals affect human movement. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating the overall process of the present invention. Detailed Implementation

[0042] The structure and working process of the present invention will be further described below with reference to the accompanying drawings.

[0043] In recent years, with the gradual development and application of machine learning algorithms, introducing intelligent analysis algorithms into the analysis of physiological signals has gradually become a new development direction in the field of rehabilitation medicine, and some rehabilitation methods based on machine learning algorithms have been proposed. The main logic of these methods is to collect one or more physiological signals, perform pattern recognition on the collected signals, determine the current action of the patient, and then drive assistive devices to compensate for the actions that the patient has not fully performed.

[0044] However, these methods often have the following key shortcomings:

[0045] 1) Almost all current methods suffer from a lag in assistive device activation. This is because current physiological signal recognition algorithms, in order to ensure accuracy, often require waiting for a certain period of time before collecting signals of sufficient duration for pattern recognition. The algorithm then analyzes the movement before initiating the assistive device's movement. This process causes significant frustration for the patient attempting the movement, hindering their rehabilitation.

[0046] 2) Not only rehabilitation methods based on machine learning algorithms, but many current rehabilitation methods ignore the differences in patients' conditions. For example, patients with milder conditions may be able to perform some simple wrist rotation and internal / external rotation movements before systematic training; while patients with more severe conditions may still be unable to control their hands or move them. If the same tasks are assigned to both types of patients and the same standards are used for assessment and pattern recognition, the training effect will inevitably be unsatisfactory.

[0047] 3) During the process of normal people performing daily actions, their corresponding physiological manifestations (such as muscle contraction, brain activity discharge, eye movement, and cardiac tremor) will exhibit a sequential nature. This sequential nature will be represented as a "sequential" signal with a time sequence during the data collection process. For example, when a subject performs the action of "vigorously swinging his arm," his motor brain area will first generate active signals, followed by the sequential discharge of the upper arm and forearm muscles, and then accompanied by an increase in cardiac frequency.

[0048] However, most current algorithms, regardless of the model used for recognition, analyze multi-lead physiological signals collected at the same time. The models themselves often fail to analyze the sequential characteristics of these signals across time dimensions. This approach loses many of the inherent features of the signals themselves, thus degrading their analysis and pattern recognition performance.

[0049] 4) Rehabilitation methods themselves belong to medical diagnosis and treatment, and this field has a certain demand for the interpretability of the algorithmic models used in these methods. Currently, rehabilitation methods that integrate machine learning algorithms often use neural networks (especially CNNs) as recognition algorithms. However, these algorithms also inherit the unavoidable "black box" drawback of most neural network algorithms, meaning it is difficult to provide a deep explanation for the correctness of the algorithm and the reasons for pattern judgment. Poor interpretability has become a major reason for the setbacks in the promotion of such algorithms.

[0050] The main technical problems solved by this invention are:

[0051] 1) Existing methods simply analyze the current physiological signals and control the movement of the assistive device based on the pattern recognition results, which leads to a delay in the activation of the assistive device and a very obvious sense of lag when using it.

[0052] Question 1: How can we activate assistive devices in advance based on existing data to achieve synchronized movement between the devices and the patient?

[0053] 2) Currently, many rehabilitation methods only design one algorithm model and do not differentiate between mild and severe patients in the algorithm. Inappropriate training methods will lead to less effective rehabilitation training and will have an adverse impact on patients' confidence in rehabilitation.

[0054] Question 2: How can we effectively incorporate the patient's current medical condition information into the methodology to make it suitable for patients with various conditions?

[0055] 3) Multiple physiological signals exhibit temporal correlation during the completion of an action, characterized as sequential signals. For patients, even erroneous feature sequences contain a wealth of information. However, current methods only analyze signals at single moments, neglecting the temporal representation of sequential signals.

[0056] Question 3: How to analyze this sequential information and extract the information contained in the signal sequence?

[0057] 4) Most existing machine learning-based rehabilitation algorithms use "black box" neural networks (such as CNNs), whose algorithm models have low interpretability and are difficult to explain in depth, which is not conducive to their promotion in the field of rehabilitation medicine.

[0058] Question 4: When designing an algorithm model, how can the interpretability of the model itself be demonstrated?

[0059] Based on the above analysis, this invention designs a rehabilitation training method for stroke patients. This method collects sequential EEG and EMG signals with sequential characteristics, maps and analyzes the collected spatiotemporal features, firstly using a sequence prediction model to predict the physiological signals of the rehabilitation patient several time frames later, and then using a graph convolutional action prediction and recognition model to identify and predict the actions the patient is about to perform. This allows assistive devices to be activated in advance, enabling them to assist the patient in completing rehabilitation actions without interruption, and better helping stroke patients restore and establish neural pathways.

[0060] In its specific implementation, this method first performs spatiotemporal mapping on the collected electromyographic signals from the patient's brain, converting the signals into graph network matrices corresponding to multiple consecutive time frames by dividing the data into windows. Then, the graph matrix data of multiple consecutive time frames is fed into a graph convolutional network (GCN) for analysis. This GCN includes an adaptive attention mechanism and a global graph structure established based on the distance between the patient's condition level and the surface electrodes, enabling spatiotemporal data prediction of the graph matrices corresponding to several future time frames. Finally, the predicted graph matrices of multiple time frames are processed by the GCN, which contains multiple fully connected layers, for pattern recognition. The analysis of the corresponding actions is used to pre-control the assistive devices to begin assisting the patient in completing the corresponding actions. This effectively and seamlessly helps the patient complete fully autonomous rehabilitation training without the need for synchronization signals.

[0061] Compared with commonly used methods, this method adds a prediction module, which predicts the characteristics of the patient's movements in the near future based on past signals. Then, it uses the features derived from real data and the predicted features together to determine what action the patient wants to take, thereby achieving a certain degree of "prediction" and activating the assistive device in advance so that the patient can complete the action without any delay, which is more conducive to the establishment of neural pathways.

[0062] To achieve the above goals, an "attention mechanism + graph network" is used for prediction. Prediction is equivalent to adding a "collection-prediction-recognition" step to the traditional "collection-recognition" process to achieve the purpose of prediction.

[0063] A method for predicting autonomous movements based on sequential spatiotemporal brain electromyography signals.

[0064] First, a spatiotemporal graph network matrix is ​​constructed to map the collected electromyographic signals of the patient's brain in a spatiotemporal manner. The signals are then converted into graph network matrices corresponding to multiple consecutive time frames by dividing the data into windows.

[0065] Then, the graph network matrix for predicting future signals is fed into the graph convolutional neural network for analysis, and spatiotemporal data prediction is achieved for the graph matrix corresponding to several future time frames.

[0066] Finally, target action recognition involves using a graph convolutional neural network with multiple fully connected layers to perform pattern recognition on the predicted multiple time frame matrices, analyzing the action corresponding to the signal, and outputting the corresponding action of the control auxiliary device in advance.

[0067] Specific embodiments, such as Figure 1 As shown.

[0068] From an overall perspective, this method can be divided into three steps in sequence: mapping, prediction, and identification. The specific details of these steps will be described below.

[0069] 1) Mapping steps:

[0070] The main function of the mapping step is to establish the spatiotemporal map matrix of the electroencephalogram (EEG) signal. By establishing the spatiotemporal map matrix, it is easier to extract the sequential features contained in the time window signal in subsequent model analysis.

[0071] In the mapping process, EEG and EMG electrodes are first used to collect the patient's EEG signals and the surface EMG signals of a specified muscle block. Then, the collected signals are divided into multiple data frames with a time window overlap rate of 20% according to their sequence. Finally, the EEG and EMG signals in each time window are spliced ​​and fused to establish a two-dimensional signal matrix, thereby obtaining a sequential two-dimensional EEG signal map of each data frame.

[0072] For ease of description later, we assume that T data frames have been divided. At this point, it is necessary to predict and identify the actions that will take place in the time window of t=T.

[0073] 2) Prediction steps:

[0074] The main function of the prediction step is to predict the signal graph matrix of the next N data frames based on the sequence signals of several given time windows, using a GCN network containing an attention self-learning graph.

[0075] During the prediction process, a signal matrix of a specified length data frame (assuming its length is L) is first selected for prediction. This data frame is then fed into the GCN for spatiotemporal data prediction in a temporal sequence. Next, this data frame sequence is fed into the GCN network to predict the next N data frames, yielding the prediction result from the GCN network, which is then used for subsequent action recognition. Internally, the GCN network module contains two parallel graph structure adjacency matrices used to perform graph convolution.

[0076] One approach combines patient condition classification with an adjacency matrix of anatomical relationships between measured brain regions and muscle blocks, based on the distance between surface electrodes. Determining matrix connectivity based on measurement distances is also a common method in the field. This matrix is ​​an empirical hyperparameter that remains fixed based on patient condition and task classification, and does not change with the increase of iterations in the early training rounds.

[0077] Secondly, there is the graph adjacency matrix obtained by multiplying the adaptive attention vectors. These two attention vectors will be iteratively trained during the model training process, just like other model parameters, so as to explore the correlation between signals in different leads within physiological features. This allows us to obtain the internal connections of sequential signals beyond the anatomical background, thereby better predicting future data frames.

[0078] Meanwhile, after completing the iterative training of the GCN model, the adjacency matrix of the self-learning attention vector also represents the adaptive attention mechanism's focus on the physiological signal map matrix itself. This mechanism is the model algorithm's understanding of the correlation between physiological representations. By comparing this adaptive matrix with the mechanism of muscle discharge in anatomy, the mechanism of action of the GCN network can be given at the correlation level within the physiological signals, thus providing an understandable explanation of its effectiveness.

[0079] 3) Identification steps:

[0080] After predicting the next N data frames, the sequence of data frames of length M, which is formed by combining the last MN data frames from the T actually acquired data frames with the predicted N data frames, can be analyzed to predict the patient's upcoming actions, where M > N. Then, based on the predicted and identified actions, the connected peripheral components are driven to assist the movement, such as mechanical gloves or electrical stimulation of corresponding muscles.

[0081] As the experimental duration increases, data frames with longer time spans will no longer be able to represent current action information. Therefore, an observation distance M is set, and pattern recognition is performed only on the sequential signal composed of the entire M data frames to analyze the corresponding action paradigm. Furthermore, to ensure that the current actual acquired signal is reflected during pattern recognition, the current data frame to the predicted data frame should be considered as a single pattern recognition process; that is, the constraint M > N should be set.

[0082] Regarding the selection of pattern recognition algorithms, this method first processes the sequential signal graph consisting of M data frames through a graph convolutional network, then expands the intermediate results after graph convolution and sends them into two consecutive fully connected layers for target action recognition.

[0083] Comparison of alternative pattern recognition algorithms in the "Identification Step"

[0084] It should be noted that if M data frames are simply spliced ​​together, a continuous multi-lead electroencephalogram (EEG) signal can be obtained. For action recognition algorithms with multiple leads, there are many alternative pattern recognition methods—such as CNN, SVM, etc., and even some traditional decision-making algorithms (such as decision trees, random forests, etc.) can achieve a certain accuracy in this task.

[0085] However, these alternative algorithms have the following shortcomings compared to the method proposed in this invention:

[0086] ① The prediction signal of this method is predicted in the form of a time window. Simply splicing the predicted signal will result in a harsh transition of the signal at the splicing point, which may cause other methods to analyze a lot of redundant information at the splicing point, which is not conducive to its recognition of action patterns.

[0087] ② This method uses graph convolution, which can transfer and utilize the empirical graph matrix in the "prediction stage" with the final generated adaptive attention graph matrix, and at the same time make the model itself more interpretable.

[0088] ③ Similar to the overall advantages of the patent, for the action recognition task of physiological signals, training with graph convolutional networks can pay more attention to the sequential features of the electromyographic signals themselves, making its performance superior to other pattern recognition algorithms under the same conditions.

[0089] Feedback on the "technical problem to be solved"

[0090] 1) The collected signals are analyzed using the GCN network to predict the signals of several future data frames. Based on the predicted signals, the current action of the patient is determined, thereby activating the assistive device in advance and achieving synchronous movement with the patient.

[0091] 2) In developing a training plan, different training tasks can be designed based on various disease conditions. For example, severely rehabilitated patients first need to train basic hand control, and their corresponding training tasks might be simple wrist raising and pressing tasks; while mildly rehabilitated patients need to train some fine motor skills, and their training tasks might include fine finger movements, drawing letters and shapes, etc. By classifying patients according to their disease conditions, and assigning different rehabilitation tasks to different conditions, this method can be applied to stroke patients at different stages of their disease.

[0092] 3) By dividing the acquired electromyographic signals into multiple data frames and establishing a graph matrix within each data frame, the analysis and prediction of sequential electromyographic signals can be achieved through the analysis of the graph matrix sequence.

[0093] 4) The model is built using a framework similar to the GCN algorithm with an adaptive graph structure. By comparing and analyzing the adjacency matrix of the attention graph after self-learning with the adjacency matrix of the empirical graph, the attention result corresponding to the algorithm model can be given for the connectivity between physiological representations. This result is easier for humans to understand than "black box" neural networks such as CNN, and naturally has better interpretability.

[0094] Examples are given below:

[0095] 1) Collection of electromyographic signals from the brain of rehabilitation patients

[0096] This method consists of a prediction model and a pattern recognition model, therefore a certain amount of synchronous electroencephalogram (EEG) signals are required to train the parameters of the two GCN models.

[0097] The data collected here mainly includes two types: first, the electroencephalogram (EEG) signals collected from rehabilitation patients during a certain paradigm movement. These signals need to be labeled and are mainly used for training the pattern recognition model; second, the EEG signals collected from rehabilitation patients wearing the acquisition device during daily rehabilitation training. These signals do not need to be labeled and are mainly used for training the prediction model. A large number of these signals need to be collected.

[0098] After data collection is complete, the data needs to be divided into data frames according to the time window. When dividing the data, a 20% overlap rate should also be followed to facilitate the training of the model to be used later.

[0099] 2) Training of the GCN sequence prediction model

[0100] The prediction model essentially analyzes the sequential features contained in the L collected data frames to predict the N future data frames. Considering the hierarchical nature of the sequential features, the prediction model requires a large amount of data. Therefore, during training, the signals from the rehabilitation training of patients need to be segmented into training data that conforms to the specified dimensions and fed into the GCN model for iterative training.

[0101] 3) Training of the graph convolutional action prediction and recognition model

[0102] Compared to predictive models, this graph convolution-based pattern recognition model has a relatively small parameter size. After performing spatiotemporal convolution and flattening on M data frames, two fully connected layers can be used to establish a mapping from intermediate result vectors to the predicted action types. During training, the graph convolution perception model needs to be fed with action-labeled information collected when patients complete rehabilitation movements to identify paradigmatic actions.

[0103] 4) Integration and adjustment of the overall model

[0104] After training both models, they can be concatenated according to the "predict first, then recognize" logic and fed with labeled real data for further training. During this process, the parameters of both models can be fine-tuned, with both the predicted sequence and the recognized action used as loss functions for the overall model. After integration and adjustments, the training of the overall model is complete.

[0105] 5) For use by stroke patients and subsequent updates

[0106] Stroke patients, after their condition level has been determined by their accompanying physician, can independently begin rehabilitation exercises while wearing the data acquisition device and assistive devices. The model based on this method will synchronously guide the patient to complete the corresponding rehabilitation exercises. Simultaneously, physiological signals from the patient's self-training can be collected for subsequent analysis and model data updates.

[0107] The autonomous action prediction device based on sequential spatiotemporal brain-myomyography signals includes a brain-myomyography signal acquisition device, a signal processing device, and auxiliary devices. The brain-myomyography signal acquisition device is used to acquire the patient's brain and muscle signals. The signal processing device is used to train a sequence prediction model and an action recognition model based on the acquired brain and muscle signals, and to issue a predicted action signal. The auxiliary devices are used to receive the predicted action signal and issue a corresponding prompt signal.

[0108] The electromyography (EMG) signal acquisition device and auxiliary devices are all wearable devices, and can be selected and worn on the corresponding parts of the patient according to the severity of the condition.

[0109] A computer storage medium storing computer-readable instructions that, when executed by a processor, invoke all or part of the steps of the method.

[0110] It should be understood that this solution is not limited to the specific embodiments described above. Devices and structures not described in detail herein should be understood as being implemented in a manner common to the art. Any person skilled in the art can make many possible variations and modifications to this solution, or modify it into equivalent embodiments, without departing from the scope of this solution, using the methods and techniques disclosed above. This does not affect the substantive content of this solution. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of this solution, without departing from its scope, still fall within the protection scope of this solution.

Claims

1. A method for predicting voluntary movements based on sequential spatiotemporal brain electromyography signals, characterized in that: First, a spatiotemporal graph network matrix is ​​constructed. Electromyography and electroencephalography signals of stroke patients are collected. By dividing the time window and splicing along the lead dimension, a spatiotemporal graph network matrix corresponding to each window time frame is established. Then, the graph network matrix for predicting future signals is obtained by feeding graph matrix data from multiple consecutive time frames into a graph convolutional neural network for analysis. By analyzing the diffusion and aggregation of signals between leads, the graph network matrix corresponding to several future time frames is predicted. Finally, for target action recognition, the predicted matrix of multiple time frames is concatenated with a portion of the actual acquired matrix of time frames to obtain an action discrimination input sequence containing longer temporal features. After inter-lead diffusion through a graph convolutional layer, it is fed into a multi-layer fully connected layer for pattern recognition to predict the action corresponding to the signal and output the corresponding action of the control auxiliary device in advance. The specific process of predicting the graph network matrix of future signals is as follows: First, a signal matrix of a specified length of data frame is selected for prediction; Then, the data frame is fed into the graph convolutional neural network in a temporal manner to perform spatiotemporal data prediction, and the prediction results of several data frames are obtained. The graph convolutional neural network includes two parallel graph structure adjacency matrices for performing graph convolution: a static adjacency matrix of the physiological association between the measured brain regions and muscle blocks, and an adaptive dynamic graph adjacency matrix obtained by multiplying the attention vectors. The static adjacency matrix of the physiological association graph between the measured brain regions and muscle blocks is a fixed empirical hyperparameter based on the patient's condition and task classification, and will not change with the increase of iterations in the early training. The dynamic graph adjacency matrix obtained by multiplying the adaptive attention vectors is continuously iterated during the model training process. The attention vectors explore the correlation between different lead signals within the physiological features and obtain the internal connections for the sequential signals themselves.

2. The method for predicting autonomous movements based on sequential spatiotemporal brain electromyography signals according to claim 1, characterized in that: The specific process of constructing the spatiotemporal graph network matrix is ​​as follows: First, collect electroencephalogram (EEG) signals and surface electromyography (EMG) signals of the designated muscles; Secondly, the collected signals are divided into multiple time windows with a certain overlap rate according to their sequence; Then, the EEG and EMG signals within each time window are spliced ​​and fused to establish a two-dimensional signal matrix, resulting in a sequential two-dimensional EEG and EMG signal map for each data frame.

3. The method for predicting autonomous movements based on sequential spatiotemporal brain electromyography signals according to claim 1, characterized in that: The specific process of target action recognition is as follows: The last MN data frames from the T data frames actually collected are combined with the predicted N data frames to form a data frame sequence of length M, which is used to predict the actions the patient is about to perform. During the prediction process, the action prediction module is used to perform pattern recognition on the sequential signal composed of the entire M data frames and analyze its corresponding action paradigm, where M>N.

4. The method for predicting autonomous movements based on sequential spatiotemporal brain electromyography signals according to claim 3, characterized in that: In the process of pattern recognition, the sequential signal graph consisting of M data frames is first passed through a graph convolutional network. Then, the intermediate result after graph convolution is expanded and sent to two consecutive fully connected layers for target action recognition.

5. The method for predicting autonomous movements based on sequential spatiotemporal brain electromyography signals according to claim 1, characterized in that: The collected electromyographic signals from the patient's brain consist of two parts. One part consists of electromyographic signals collected during a rehabilitation patient's performance of a certain paradigmatic movement. These signals need to be labeled for training the pattern recognition model. The other part consists of electromyographic signals collected during daily rehabilitation training while the rehabilitation patient wears the acquisition device. These signals do not need to be labeled and are used for training the prediction model. A large number of these signals need to be collected.

6. The method for predicting autonomous movements based on sequential spatiotemporal brain electromyography signals according to claim 5, characterized in that: After the pattern recognition model and the prediction model are trained, the two models are concatenated according to the logic of "predict first and then recognize", and trained using real data with labels. During the training process, the parameters of the two models are fine-tuned, and both the predicted sequence and the recognized action are used as the loss function of the overall model.

7. A device for predicting autonomous movements based on sequential spatiotemporal brain electromyography signals according to any one of claims 1 to 6, characterized in that: It includes a brain and muscle electromyography (BEM) signal acquisition device, a signal processing device, and auxiliary devices; wherein, the BEM signal acquisition device is used to acquire the patient's brain and muscle electromyography (BEM) signals; the signal processing device is used to train a sequence prediction model and a motion recognition model based on the acquired BEM and muscle electromyography (BEM) signals, and to issue a predicted motion signal; the auxiliary devices are used to receive the predicted motion signal and issue a corresponding prompt signal.

8. The autonomous action prediction device based on sequential spatiotemporal brain electromyography signals according to claim 7, characterized in that: The electromyography (EMG) signal acquisition device and auxiliary devices are all wearable devices, and can be selected and worn on the corresponding parts of the patient according to the severity of the condition.

9. A computer storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions that, when executed by a processor, invoke all or part of the steps of the method according to any one of claims 1 to 6.

Images