Motion intention recognition method based on dual-stream Transformer encoder and multi-head attention mechanism

Abstract

The present invention discloses a method for identifying motion intention based on a dual-stream Transformer encoder and a multi-head attention mechanism, comprising: obtaining multiple groups of sample information collected by a wearable device worn by a stroke patient, each group of sample information comprising a sample electromyographic signal, an inertial measurement signal and / or a sample electroencephalogram signal; establishing a motion intention identification model based on a dual-stream Transformer encoder and a multi-head attention mechanism based on each group of sample information; determining the motion intention of the stroke patient based on the motion intention identification model; wherein the motion intention identification network comprises a dual-stream Transformer encoder, a long-short sequence feature cross-attention module, a multi-scale feature fusion module and a motion intention classification module; the dual-stream Transformer encoder comprises a multi-head attention mechanism; also disclosed are a motion intention identification system based on the dual-stream Transformer encoder and the multi-head attention mechanism, application of the motion intention identification method in mirror therapy and / or assisted movement therapy for stroke patients, an electronic device and a computer-readable storage medium.

Description

Technical Field

[0001] The present invention relates to the field of computer virtual reality and intelligent rehabilitation technology, and in particular to a motion intention recognition method based on a dual-stream Transformer encoder and a multi-head attention mechanism. Background Art

[0002] "Stroke", also known as "apoplexy" or "cerebrovascular accident", is an acute cerebrovascular disease. It is a group of diseases that cause brain tissue damage due to sudden rupture of brain blood vessels or blood vessel blockage that prevents blood from flowing into the brain, including ischemic and hemorrhagic strokes. The incidence of ischemic stroke is higher than that of hemorrhagic stroke, accounting for 60% to 70% of the total number of strokes. Occlusion and stenosis of the internal carotid artery and vertebral artery can cause ischemic stroke. Most patients are over 40 years old, and more men than women. Severe cases can cause death. The mortality rate of hemorrhagic stroke is higher. Surveys show that urban and rural stroke has become the number one cause of death in my country and the leading cause of disability among Chinese adults. Stroke has the characteristics of high incidence, high mortality and high disability rate. Among them, wrist varus is a common clinical manifestation of stroke. Patients have atrophy of arm muscles and lose the grasping function of their hands, which is very inconvenient.

[0003] Movement intention is the key to accurately track the movement of human upper limbs and ultimately achieve mirror therapy of upper limbs. Although the research on movement intention in the field of exoskeleton has achieved certain results in recent years, the technology is still not mature enough. The key to movement intention recognition is to obtain the time series of myoelectric signals and other signals of incomplete movements performed at the current moment for stroke patients, analyze the patient's expected movements from them, and guide the affected hand to perform rehabilitation exercises according to the patient's movement intention. At present, the recognition methods for human movement intention mainly include intention recognition based on mechanical information and intention recognition based on bioelectric information. However, the movement intention recognition method using mechanical information has a serious lag because it can only be obtained after the user starts to exercise, and it cannot directly reflect the person's movement intention, making it difficult to achieve flexible control. Since continuous movement of the human body can cause problems such as decreased muscle contractility and sweating on the upper epidermis, the accuracy of the prediction results of movement intention is reduced. Intention recognition based on bioelectric information needs to fully consider the impact of the user's muscle state on myoelectric information after long-term use. Therefore, people have begun to study the application of machine learning methods in the field of movement intention recognition. For the research on the recognition of human body movement intention, multiple sensors such as acceleration, angular velocity, and pressure are usually worn on the human body and exoskeleton to collect physiological signals, and then predict the form of human movement, so as to control the movement of the exoskeleton robot. For the research on the field of human lower limb rehabilitation movement, for example: Marion and other scholars studied the use of feedforward neural networks and long short-term memory neural networks to predict the ground reaction force and joint torque during lower limb movement, with an average prediction accuracy of 0.95, see the literature "Mundt Marion, Koeppe Arnd, David Sina, Bamer Franz, Potthast Wolfgang, Markert Bernd. Predict ion of ground reaction force and joint moments based on optical motion capture data during gait. [J]. Medical engineering & physics, 2020, 86.". However, due to the simple network structure of the above method, the feature extraction effect of bioelectric signals at obvious changes is poor, so the prediction accuracy is generally low.

[0004] Therefore, the prior art does not have a mature and applicable technical solution for using recognition models to identify movement intentions in mirror rehabilitation therapy and moderate and mild assisted movement therapy for severe stroke patients. Summary of the invention

[0005] In order to solve the problems existing in the prior art, the present invention provides the following motion intention recognition method based on a dual-stream Transformer encoder and a multi-head attention mechanism. Based on the sample signals of the patient under different actions collected, an electromyographic motion intention recognition model is established after data sample expansion. Based on the recognition model, the motion intention can be identified with a high accuracy, thereby providing a basis for effective active rehabilitation treatment of stroke patients.

[0006] The object of the present invention is to provide a motion intention recognition method based on a dual-stream Transformer encoder and a multi-head attention mechanism, comprising:

[0007] S1, obtaining multiple groups of sample information collected by a wearable device worn by a stroke patient, each group of sample information including a sample electromyographic signal, an inertial measurement signal and / or a sample electroencephalographic signal;

[0008] S2, based on each group of sample information, establishes a motion intention recognition model based on a dual-stream Transformer encoder and a multi-head attention mechanism;

[0009] S3, determining the movement intention of the stroke patient based on the movement intention recognition model;

[0010] Wherein, the S2, establishing a motion intention recognition model based on a dual-stream Transformer encoder and a multi-head attention mechanism based on each set of sample information, includes:

[0011] S21, preprocessing the sample information and obtaining a first part of data set data required for establishing a motion intention recognition model;

[0012] S22, expanding the first part of the data set to obtain the second part of the data set, and merging the first part of the data set with the second part of the data set to form a sample data set;

[0013] S23, establishing a motion intention recognition network based on a dual-stream Transformer encoder and a multi-head attention mechanism; the motion intention recognition network includes a dual-stream Transformer encoder, a long-short sequence feature cross attention module, a multi-scale feature fusion module and a motion intention classification module; the dual-stream Transformer encoder includes a multi-head attention mechanism;

[0014] S24, inputting the sample training data set into the motion intention recognition network for training and learning to obtain the motion intention recognition model.

[0015] Preferably, the wearable device is an electromyography acquisition sensor, an inertial measurement sensor and / or an electroencephalography acquisition sensor.

[0016] Preferably, the preprocessing of S21 includes noise reduction, normalization, taking absolute value and data segmentation, wherein:

[0017] S211, the noise reduction includes filtering out power frequency interference, motion artifacts and / or noise caused by multi-channel crosstalk in the original electromyographic signal, thereby obtaining sample information after the noise is filtered out;

[0018] S212, the normalization includes limiting the sample information after the noise is filtered out to a size that is conducive to model training, so as to obtain a normalized electromyographic signal;

[0019] S213, taking the absolute value includes: taking the absolute value of all sequences of the normalized electromyographic signals;

[0020] S214, data segmentation: cutting the entire sequence of normalized electromyographic signals after taking absolute values ​​into a plurality of sample time series windows, and using the plurality of sample time series windows as data set data.

[0021] Preferably, the step S22 of expanding the first part of the data set to obtain the second part of the data set includes:

[0022] S221, random windowing: randomly windowing the data of the first part of the data set to obtain randomly windowed sample data in the data of the second part of the data set, including: randomly selecting a starting point of a window in each type of action sequence, and determining an end point according to the window length to obtain an electromyographic timing window; based on the electromyographic timing window, randomly windowing the entire sequence of normalized sample information after taking the absolute value to obtain the randomly windowed sample data in the data of the second part of the data set;

[0023] S222, delay signal enhancement: performing delay signal enhancement on the data of the first part of the data set to obtain the delay signal enhanced sample data in the data of the second part of the data set, including: randomly selecting a section of sampling points of one of the multiple sample timing windows of S214 and deleting it; selecting the sampling points of the next moment of one of the sample timing windows with the same number of deleted sampling points and putting them into the end of the window to form a delay signal enhancement timing window; performing delay signal enhancement on the entire sequence of normalized sample information after taking the absolute value based on the delay signal enhancement timing window to obtain the delay signal enhanced sample data in the data of the second part of the data set;

[0024] S223, merging the randomly windowed sample data in the second part of the data set and the delayed signal enhanced sample data in the second part of the data set to obtain the second part of the data set; merging the first part of the data set with the second part of the data set to form a sample data set based on data enhancement merging, thereby effectively expanding the data volume of the sample data set.

[0025] Preferably, the S22 only includes executing the random windowing in S221 or the delayed signal enhancement in S222, and accordingly chooses not to implement S223.

[0026] Preferably, the dual-stream Transformer encoder includes a channel attention module, a long sequence slice transformation module, a short sequence slice transformation module, a multi-head attention mechanism module and a feedforward neural network module;

[0027] The long-short sequence feature cross attention module is used to simultaneously learn the identification information of the long sequence branch and the short sequence branch obtained by the long sequence slice transformation module and the short sequence slice transformation module;

[0028] The multi-scale feature fusion module is used to fuse the recognition information of the long sequence branch and the recognition information of the short sequence branch through the long-short sequence feature cross attention module and output a multi-scale fusion feature;

[0029] The motion intention classification module is used to use full connection to perform motion intention classification on the multi-scale fusion features to obtain a motion intention output result.

[0030] Preferably, establishing the dual-stream Transformer encoder includes:

[0031] S231, establishing a channel attention module, including: calculating the connection between each channel according to the temporal characteristics of the sample information and the spatial characteristics of the multi-channel electromyographic signal, learning the importance of the signal characteristics of each channel in the spatial distribution to the rehabilitation movement recognition, and adaptively adjusting the recognition weight of each channel, so that the sample timing window passing through the channel attention module forms the channel attention;

[0032] S232, establishing a long sequence slicing transformation module and a short sequence slicing transformation module, including: slicing the sample time sequence window forming the channel attention according to the number of sampling points within a certain period of time to form a long sequence with more sampling points and a short sequence with fewer sampling points; transforming the long sequence and the short sequence of each slice into a long sequence slice one-dimensional vector and a short sequence slice one-dimensional vector through a long sequence slicing module and a short sequence slicing module respectively;

[0033] S233, establish a multi-head attention mechanism module;

[0034] S234, establish a feedforward neural network, which is composed of multiple fully connected layers. A first residual connection and normalization module is set between the feedforward neural network and the multi-head attention mechanism module; a second residual connection and normalization module is set between the feedforward neural network and the long-short sequence feature cross-attention module.

[0035] Preferably, the calculation formula of each head of the multi-head attention mechanism in S233 is as shown in formula (1). The calculation formula of the attention mechanism can be used to learn the correlation between each slice in the same layer:

[0036]

[0037] Among them, Attention(Q, K, V) is a multi-head attention mechanism, Q, K, V are query matrix, key matrix and value matrix respectively, d is the row vector dimension of the matrix, Softmax is a normalized exponential function, the Softmax function is a single-layer neural network, which is a generalization of the binary classification function sigmoid in multi-classification, and is used to display the results of multi-classification in the form of probability. The calculation method of Softmax is: Softmax converts the prediction results from negative infinity to positive infinity into probabilities in two steps: the first step is to convert the prediction results of the model to the exponential function to ensure the non-negativity of the probability; the second step is to ensure that the sum of the probabilities of each prediction result is equal to 1, and the converted results are normalized, that is, the converted results are divided by the sum of all converted results, and the percentage of the converted results in the total is used as the approximate probability.

[0038] Preferably, the multi-head attention mechanism module includes:

[0039] A multi-head slice formation module, used for constructing a long sequence slice matrix and a short sequence slice matrix based on the long sequence slice one-dimensional vector and the short sequence slice one-dimensional vector and inputting them into the slice matrix transformation module;

[0040] Slice matrix transformation module: used to linearly transform the long sequence slice matrix and the short sequence slice matrix to obtain the query matrix Q, the key matrix K and the value matrix V, and obtain the transformed query matrix Q', the key matrix K' and the value matrix V' through the fully connected layer and input them into the single-head attention transformation module;

[0041] Single-head attention transformation module: used to obtain multiple single-head attention matrices based on the transformed query matrix Q', key matrix K' and value matrix V';

[0042] Multi-head attention fusion module: used to concatenate multiple single-head attention matrices to construct a multi-head attention matrix, compress it, and output the compressed multi-head attention matrix X' or Y';

[0043] The multi-head slice forming module is used to receive n short sequence slice one-dimensional vectors a output by the short sequence slice module 1 ,a 2 ,…,a n , or receive the L long sequence slice one-dimensional vector b output by the long sequence slice module1 ,b 2 ,…,b L ; Construct short sequence slice matrix X = [a 1 ,a 2 ,…,a n ] or long sequence slice matrix Y = [b 1 ,b 2 ,…,b L ]; output the short sequence slice matrix X or the long sequence slice matrix Y to the slice matrix transformation module.

[0044] The slice matrix transformation module is used to receive the short sequence slice matrix X or the long sequence slice matrix Y, and obtain the query matrix Q, the key matrix K and the value matrix V through linear transformation, as shown in equations (11), (12) and (13);

[0045] For a short sequence slice matrix X, then:

[0046] Q=W q X+b q (11);

[0047] K=W k X+b k (12);

[0048] V=W v X+b v (13);

[0049] Or perform the same operations as equations (11), (12), and (13) for the long sequence slice matrix Y;

[0050] Where W q , W K , W v , is the matrix with learnable parameters in each attention mechanism, b q , b k , b v It is the matrix bias, which updates the parameter matrix and matrix bias through optimization during model training;

[0051] After obtaining the Q, K, V values, the converted query matrix Q', key matrix K' and value matrix V' are obtained through the fully connected layer, and the converted query matrix Q', key matrix K' and value matrix V' are output to the single-head attention transformation module to construct a single head of the multi-head attention mechanism;

[0052] The single-head attention transformation module is used to receive the transformed query matrix Q', key matrix K' and value matrix V' output by the slice matrix transformation module; and then perform the following processing: first, transpose the transformed key matrix K' and dot multiply it with the transformed query matrix Q'; then divide the dot product result by the power of half of the row vector dimension d of the matrix Finally, the above calculation results are normalized by the Softmax function and multiplied by the converted value matrix V' to obtain the output matrix head containing the single-head attention information;

[0053] The calculation formula of the single-head attention is as follows (1'):

[0054]

[0055] Where head is the output matrix containing single-head attention information, Attention(Q',K',V') is the single-head attention transformation, Q', K', V' are the query matrix, key matrix and value matrix after transformation, Q', K', V' have the same matrix dimensions, and d is the row vector dimension of the matrix;

[0056] When there are h single-head attention modules, h single-head attention output matrices head 1, head 2, ..., head i, ..., head h are obtained respectively, as shown in formula (14):

[0057] in

[0058] And output the h single-head attention matrices to the multi-head attention fusion module;

[0059] The multi-head attention fusion module is used to receive the h single-head attention output matrices head 1, head 2, ..., head i, ..., head h, and concatenate them to construct a matrix containing multi-head attention information, as described in formula (15):

[0060] MultiHead(Q',K',V')=concat(head 1,...,head h) (15);

[0061] The concat function can connect multiple matrices along the specified axis to form a concatenated matrix;

[0062] After compressing the multi-head attention matrix MultiHead(Q', K', V') to obtain the compressed multi-head attention matrix X' or Y', the compressed multi-head attention matrix X' or Y' is output to the first residual connection and normalization module for processing, wherein the compression processing includes: using a fully connected layer to compress the multi-head attention matrix so that the compressed multi-head attention matrix is ​​consistent with the dimension of the single-head attention matrix.

[0063] A second aspect of the present invention provides a motion intention recognition system based on a dual-stream Transformer encoder and a multi-head attention mechanism, comprising:

[0064] A sample collection module, used to obtain multiple groups of sample information collected by a wearable device worn by a stroke patient, each group of sample information including a sample electromyographic signal, an inertial measurement signal and / or a sample electroencephalogram signal;

[0065] The model building module is used to build a motion intention recognition model based on a dual-stream Transformer encoder and a multi-head attention mechanism based on each set of sample information;

[0066] A movement intention recognition module is used to determine the movement intention of the stroke patient based on the movement intention recognition model.

[0067] The third invention of the present invention provides an application of a movement intention recognition method based on a dual-stream Transformer encoder and a multi-head attention mechanism in mirror therapy and / or assisted movement therapy for stroke patients.

[0068] A fourth aspect of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores a plurality of instructions, and the processor is used to read the instructions and execute the method as described in the first aspect.

[0069] A fifth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of instructions, and the plurality of instructions can be read by a processor and execute the method described in the first aspect.

[0070] The motion intention recognition method, system, application, electronic device and computer-readable storage medium based on the dual-stream Transformer encoder and multi-head attention mechanism provided by the present invention have the following beneficial technical effects:

[0071] The sample signals of the patient's wearable device are collected, and after preprocessing, all the signals are cut into sample time windows as data sets, and the data sets are enhanced to expand the training samples. The sample signals are sliced ​​into long and short sequences and input into the dual-stream Transformer encoder respectively to fully extract the information of the long and short time series of the signal. The extracted features are used with the long-short sequence feature cross-attention module and the multi-scale feature fusion module, taking into account the long and short time series information in the sample time window, and feature fusion is performed to finally form a motion intention recognition network based on the long-short time cross-attention dual-stream multi-scale fusion. Finally, the patient's motion intention is obtained through the intention classification module to achieve high-accuracy motion intention recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are certain embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0073] Figure 1(a) is a flow chart of the motion intention recognition method based on a dual-stream Transformer encoder and a multi-head attention mechanism described in the present invention; Figure 1(b) is a flow chart of the method for establishing a motion intention recognition model based on a dual-stream Transformer encoder and a multi-head attention mechanism based on each group of sample information described in the present invention.

[0074] Figure 2 This is a schematic diagram of the motion intention recognition network structure based on the dual-stream Transformer encoder and multi-head attention mechanism described in the present invention.

[0075] Figure 3 This is a schematic diagram of the principle architecture of the multi-head attention mechanism module described in the present invention.

[0076] Figure 4 This is a schematic diagram of the module corresponding to the multi-head attention mechanism described in the present invention.

[0077] Figure 5 This is a schematic diagram of the principle architecture of the motion intention recognition system based on the dual-stream Transformer encoder and multi-head attention mechanism described in the present invention.

[0078] Figure 6 It is a schematic diagram of the structure of the electronic device described in the present invention. DETAILED DESCRIPTION

[0079] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0080] The method provided by the present invention can be implemented in the following terminal environment, and the terminal may include one or more of the following components: a processor, a memory, and a display screen. The memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the method described in the following embodiment.

[0081] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect various parts of the entire terminal, and executes various functions of the terminal and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory.

[0082] The memory may include random access memory (RAM) or read-only memory (ROM). The memory may be used to store instructions, programs, codes, code sets or instructions.

[0083] The display screen is used to display the user interface of each application.

[0084] In addition, those skilled in the art can understand that the structure of the above terminal does not constitute a limitation on the terminal, and the terminal may include more or fewer components, or combine certain components, or arrange the components differently. For example, the terminal also includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, and a power supply, which will not be described in detail here.

[0085] Embodiment 1

[0086] Referring to FIG. 1(a), a motion intention recognition method based on a dual-stream Transformer encoder and a multi-head attention mechanism includes:

[0087] S1, obtaining multiple groups of sample information collected by a wearable device worn by a stroke patient, each group of sample information including a sample electromyographic signal, an inertial measurement signal and / or a sample electroencephalographic signal;

[0088] S2, based on each group of sample information, establishes a motion intention recognition model based on a dual-stream Transformer encoder and a multi-head attention mechanism;

[0089] S3, determining the movement intention of the stroke patient based on the movement intention recognition model;

[0090] See Figure 1(b) and Figure 2 , wherein said S2, based on each set of sample information, establishes a motion intention recognition model based on a dual-stream Transformer encoder and a multi-head attention mechanism, including:

[0091] S21, preprocessing the sample information and obtaining a first part of data set data required for establishing a motion intention recognition model;

[0092] S22, expanding the first part of the data set to obtain the second part of the data set, and merging the first part of the data set with the second part of the data set to form a sample data set;

[0093] S23, establishing a motion intention recognition network based on a dual-stream Transformer encoder and a multi-head attention mechanism; the motion intention recognition network includes a dual-stream Transformer encoder, a long-short sequence feature cross attention module, a multi-scale feature fusion module and a motion intention classification module; the dual-stream Transformer encoder includes a multi-head attention mechanism;

[0094] S24, combining the data of the first data set and the data of the second data set into a sample training data set, and inputting the sample training data set into the motion intention recognition network for training and learning to obtain the motion intention recognition model.

[0095] As a preferred embodiment, the wearable device is an electromyography sensor, an inertial measurement sensor and / or an electroencephalography sensor. In this embodiment, the wearable device is an electromyography sensor, which is fixed at a corresponding position of the healthy hand of the stroke patient.

[0096] As a preferred implementation, the preprocessing of S21 includes noise reduction, normalization, taking absolute value and data segmentation, wherein:

[0097] S211, the noise reduction includes: setting a filter type and coefficient and a blind source separation method according to the type of the sample information; filtering out the noise caused by power frequency interference, motion artifacts and / or multi-channel crosstalk in the original electromyographic signal based on the filter and the blind source separation method, thereby obtaining the sample information after the noise is filtered out;

[0098] S212, the normalization includes: limiting the sample information after the noise is filtered out to a size that is conducive to model training based on a Z-score or maximum and minimum normalization method to obtain a normalized electromyographic signal;

[0099] S213, taking the absolute value includes: taking the absolute value of the entire sequence of the normalized electromyographic signal; the implementation of step S213 is because in the normalized electromyographic signal, the signal amplitude of each action segment is positive or negative, but both positive and negative can represent muscle contraction, and not performing absolute value processing may offset useful information.

[0100] S214, data segmentation: cutting the entire sequence of normalized sample information after taking the absolute value into multiple sample timing windows, and using the multiple sample timing windows as data set data. In this embodiment, the sample data is an electromyographic signal, which is a time series. According to the sampling rate of the sampling device, a long sequence will be collected. Since the data is too long, the data cannot be trained and recognized. Therefore, the entire sequence of normalized electromyographic signals after taking the absolute value is cut into multiple electromyographic timing windows, that is, taking windows, recorded as original electromyographic timing windows, and the original electromyographic timing windows are used as data set data. The data set data is subsequently enhanced and merged to obtain training data for training the motion intention recognition model and test data for the test model.

[0101] As a preferred implementation, in S22, expanding the first part of the data set to obtain the second part of the data set includes randomly windowing and delay signal enhancing the first part of the data set to obtain randomly windowed sample data in the second part of the data set and delay signal enhanced sample data in the second part of the data set, and merging the randomly windowed sample data in the second part of the data set and the delay signal enhanced sample data in the second part of the data set to obtain the second part of the data set.

[0102] As a preferred implementation, the step S22 of expanding the first part of the data set to obtain the second part of the data set includes:

[0103] S221, random windowing: including: randomly selecting the starting point of the window in each type of action sequence, and determining the end point according to the window length to obtain the electromyographic timing window; based on the electromyographic timing window, randomly windowing the entire sequence of normalized sample information after taking the absolute value to obtain the randomly windowed sample data in the second part of the data set; for the electromyographic signal in this embodiment, the purpose of the S221 operation is to obtain the electromyographic timing window that cannot be obtained during the implementation of S214, so as to increase the sample diversity of the data set.

[0104] S222, delay signal enhancement: including: randomly selecting a section of sampling points from one of the multiple sample timing windows of S214 and deleting it; selecting the sampling points with the same number of deleted sampling points at the next moment of one of the sample timing windows and putting them into the end of the window to form a delay signal enhancement timing window; based on the delay signal enhancement timing window, performing delay signal enhancement on the entire sequence of normalized sample information after taking the absolute value to obtain the delay signal enhancement sample data in the second part of the data set. In this embodiment, the purpose of implementing S222 is that the electromyography acquisition sensor may have internal components or data omissions during transmission and reception, which can increase the sample volume and robustness of the system;

[0105] S223: Combine the randomly windowed sample data in the second part of the data set and the delayed signal enhanced sample data in the second part of the data set to obtain the second part of the data set.

[0106] The merging of the first part of the data set with the second part of the data set to form the sample data set is implemented based on data enhancement merging, thereby effectively expanding the data volume of the sample data set.

[0107] It should be noted that S22 only includes executing the random windowing of S221 or the delayed signal enhancement of S222, and accordingly chooses not to implement S223. Therefore, the above three data amplification methods (including only S221, only S222, and including S221-S223 at the same time) are all within the protection scope of the present invention, and can effectively expand the data volume of the sample data set to varying degrees.

[0108] See again Figure 2 , where "XM" and "XN" indicate that the structure in the dotted line is repeated M times and N times to construct a deep encoder and extract deeper features. Based on this, the long-short sequence feature cross attention module is used to simultaneously learn the recognition information of the long sequence branch and the short sequence branch obtained by the long sequence slice transformation module and the short sequence slice transformation module respectively; specifically, in this embodiment, the recognition information and Figure 2 The short sequence features shown in correspond to the long sequence features, so the long-short sequence feature cross attention module is used to simultaneously learn the long sequence features output by the long sequence branch constructed by the long sequence slice transformation module and the short sequence slice transformation module and the short sequence features output by the short sequence branch;

[0109] The multi-scale feature fusion module is used to fuse the recognition information learned by the long sequence branch of the long-short sequence feature cross-attention module and the recognition information learned by the short sequence branch, and then output a multi-scale fusion feature; in this embodiment, the specific method is to first use the CLS token (classification token, classification slice) of each branch as an agent to exchange information between the patch tokens (sequence slices) of another branch, and then project it to its own branch. Since the CLS token has learned the abstract information between all patch tokens in its own branch, the interaction with the patch token in another branch helps to fuse information of different scales. After fusing with other branch tokens, the CLS token interacts with its own patch token again on the next layer of Transformer encoder. In this step, it can pass the learning information from another branch to its own patch token to enrich the feature representation of each patch token;

[0110] The motion intention classification module uses full connection to perform motion intention classification on the multi-scale fusion features to obtain a motion intention output result.

[0111] As a preferred implementation, the dual-stream Transformer encoder includes a channel attention module, a long sequence slice transformation module, a short sequence slice transformation module, a multi-head attention mechanism module and a feedforward neural network module.

[0112] Building the dual-stream Transformer encoder includes:

[0113] S231, establishing a channel attention module, including: calculating the connection between each channel according to the temporal characteristics of the sample information and the spatial characteristics of the multi-channel electromyographic signal, in this embodiment, one channel corresponds to one electromyographic sensor, and multiple electromyographic sensors are distributed on different muscle groups; learning the importance of the signal characteristics of each channel in the spatial distribution to the rehabilitation movement recognition, and adaptively adjusting the recognition weight of each channel, so that the sample timing window passing through the channel attention module forms a channel attention; thereby enabling the movement intention recognition network to better extract the movement information contained in the electromyography;

[0114] S232, establish a long sequence slicing transformation module and a short sequence slicing transformation module, including: slicing the sample time sequence window that forms the channel attention according to the number of sampling points within a certain period of time, forming a long sequence with more sampling points and a short sequence with fewer sampling points respectively; transforming the long sequence and the short sequence of each slice into a long sequence slice one-dimensional vector and a short sequence slice one-dimensional vector respectively through a long sequence slicing module and a short sequence slicing module; in this embodiment, the collected electromyographic signal contains rehabilitation gestures of various difficulty levels set according to the different degrees of mobility of the affected hand. Complex rehabilitation gestures are more dependent on the changing features in the long-term electromyographic sequence, and simple rehabilitation gestures are more dependent on the changing features in the short-term electromyographic sequence; and extracting the features of the long-term electromyographic sequence will cause information redundancy for the recognition of simple gestures, and extracting the feature information of the short-term electromyographic sequence is not enough to identify complex rehabilitation gestures, so slicing long sequences and short sequences at the same time is beneficial to the recognition of various gestures;

[0115] S233, establish a multi-head attention mechanism module.

[0116] See also Figure 3 , the multi-head attention mechanism module consists of:

[0117] Multi-head slicing module S1: used to receive the n short sequence slice one-dimensional vector a output by the short sequence slicing module 1 ,a 2 ,…,a n , or receive the L long sequence slice one-dimensional vector b output by the long sequence slice module 1 ,b 2 ,…,b L . And construct the short sequence slice matrix X = [a 1 ,a 2 ,…,a n ] or long sequence slice matrix Y = [b 1 ,b 2 ,…,b L ]; output the short sequence slice matrix X or the long sequence slice matrix Y to the slice matrix transformation module S2;

[0118] Slice matrix transformation module S2: receives the short sequence slice matrix X or the long sequence slice matrix Y, and obtains the query matrix Q, the key matrix K and the value matrix V through linear transformation, as shown in equations (11), (12) and (13);

[0119] For a short sequence slice matrix X, then:

[0120] Q=W q X+b q (11);

[0121] K=W k X+bk (12);

[0122] V=W v X+b v (13);

[0123] For the long sequence slice matrix Y, perform the same operations as equations (11), (12), and (13);

[0124] Where W q , W K , W v , is the matrix with learnable parameters in each attention mechanism, b q , b k , b v For matrix bias, during the model training process, the parameter matrix and matrix bias are optimized and updated to make the model output close to the correct motion intention;

[0125] After obtaining the Q, K, and V values, the transformed query matrix Q', key matrix K', and value matrix V' are obtained through the fully connected layer, and the transformed query matrix Q', key matrix K', and value matrix V' are output to the single-head attention transformation module S3 to construct a single head of the multi-head attention mechanism;

[0126] Single-head attention module S3: used to receive the transformed query matrix Q', key matrix K' and value matrix V' output by the slice matrix transformation module S2; first transpose the key matrix K' and dot multiply it with the matrix Q'; then divide the dot product result by the power of half of the matrix row vector dimension d The purpose is to reduce the value of matrix parameters to facilitate model calculation; finally, the above calculation results are normalized by the Softmax function and multiplied by the value matrix V' to obtain the output matrix head containing single-head attention information;

[0127] The calculation formula of the single-head attention module S3 is as follows: Figure 3 The dotted line part of S3 in the figure. The correlation between slices can be learned through the calculation formula of the single-head attention module.

[0128]

[0129] Where head is the output matrix containing single-head attention information, Attention(Q',K',V') is the single-head attention transformation, Q', K', V' are the query matrix, key matrix and value matrix after transformation, Q', K', V' have the same matrix dimensions, and d is the row vector dimension of the matrix.

[0130] When there are h single-head attention modules, h single-head attention output matrices head 1, head 2, ..., head i, ..., head h are obtained respectively, as shown in formula (14)

[0131] in

[0132] And the h single-head attention matrices are output to the multi-head attention fusion module S4 to obtain short sequence slices X' and long sequence slices Y' with attention information.

[0133] The multi-head attention fusion module S4 includes the following functions:

[0134] (1) Multi-head attention concatenation: Receive the h single-head attention output matrices head 1, head 2, …, headi, …, head h, and concatenate them to construct a matrix containing multi-head attention information, as described in formula (15). Through multi-head attention concatenation, attention information of multiple dimensions can be effectively integrated.

[0135] MultiHead(Q',K',V')=concat(head 1,...,head h) (15);

[0136] The concat function can connect multiple matrices along the specified axis to form a concatenated matrix;

[0137] (2) Compressing the multi-head attention matrix MultiHead(Q', K', V') to obtain the compressed multi-head attention matrix X' or Y'; the compression process includes: using a fully connected layer to compress the multi-head attention matrix so that the dimension of the compressed multi-head attention matrix is ​​consistent with that of the single-head attention matrix.

[0138] Since increasing the network depth can extract more detailed features when building a model using the attention model, it is an effective way to improve model performance. As described above, the calculation formula (1) for each head of the multi-head attention mechanism in Transformer is as follows:

[0139]

[0140] Among them, Attention(Q,K,V) is a multi-head attention mechanism, Q, K, V are query matrix, key matrix and value matrix respectively, d is the dimension of matrix row vector, Softmax is a normalized exponential function, Softmax function is a single-layer neural network, which is the generalization of binary classification function sigmoid in multi-classification, and is used to display the results of multi-classification in the form of probability. The calculation method of Softmax is: Softmax converts the prediction results from negative infinity to positive infinity into probabilities in two steps: the first step is to convert the prediction results of the model to the exponential function to ensure the non-negativity of the probability; the second step is to ensure that the sum of the probabilities of each prediction result is equal to 1, and the converted results are normalized, that is, the converted results are divided by the sum of all converted results, which can be understood as the percentage of the converted results in the total number, and the approximate probability is obtained;

[0141] (3) Outputting the compressed multi-head attention matrix X' or Y' to the first residual connection and normalization module for processing, wherein the establishment of the first residual connection and normalization module is described in detail below.

[0142] S234, establishing a feedforward neural network, wherein the feedforward neural network is composed of a plurality of fully connected layers, a first residual connection and a normalization module are set between the feedforward neural network and the multi-head attention mechanism module; a second residual connection and a normalization module are set between the feedforward neural network and the long-short sequence feature cross attention module;

[0143] The two residual connection and normalization modules are used to solve the problem of multi-layer neural network training. They are used to weight the input and output of the previous module and perform normalization. The two modules are used to effectively transmit shallow information to deep layers, thereby effectively solving the problem of gradient disappearance. It includes two parts: residual connection Add and normalization Norm:

[0144] (1) Residual connection Add stands for Residual Connection. It improves model performance and solves the problem of difficulty in training multi-layer neural networks by transferring part of the information of the previous layer to the next layer without difference. For some layers, it is impossible to determine whether their effect is positive. After adding residual connection, the information of the previous layer is divided into two paths. One part is changed through the layer, and the other part is directly passed to the next layer. The results of the two parts are added as the input of the next layer, so that at least the information of the previous layer can be retained after the residual connection;

[0145] (2) Normalization Norm is layer normalization, which accelerates the model training process by normalizing the activation values ​​of the layer, thereby achieving a faster convergence speed. In this embodiment, normalization includes two methods: normalizing the same feature of different samples in the same batch or normalizing different features of the same sample in the channel direction.

[0146] Embodiment 2

[0147] See also Figure 5 Embodiment 2 provides a motion intention recognition system based on a dual-stream Transformer encoder and a multi-head attention mechanism, including:

[0148] The sample collection module 101 is used to obtain multiple groups of sample information collected by the wearable device worn by the stroke patient, each group of sample information includes a sample electromyographic signal, an inertial measurement signal and / or a sample electroencephalogram signal;

[0149] A model building module 102 is used to build a motion intention recognition model based on a dual-stream Transformer encoder and a multi-head attention mechanism based on each set of sample information;

[0150] The movement intention recognition module 103 is used to determine the movement intention of the stroke patient based on the movement intention recognition model.

[0151] The third invention of the present invention provides an application of a motion intention recognition method based on a dual-stream Transformer encoder and a multi-head attention mechanism in mirror therapy and / or assisted movement therapy for stroke patients. Among them:

[0152] (I) Severely ill patients

[0153] Applicable subjects: patients with Brunnstorm stage I (no voluntary movement (slow stage)) and II (only very slight flexion).

[0154] Patient characteristics: There is no voluntary movement or only slight bending of the affected hand, and there is no reliable electromyographic signal to express movement intention.

[0155] Measures: Mirror therapy, collecting electromyographic signals of the healthy hand to identify movements, thereby controlling the movement of the rehabilitation assistive device of the affected hand.

[0156] (II) Moderate patients

[0157] Applicable to: Brunnstorm stage III (whole body grasp, use hook grasp but cannot relax, cannot extend fingers), IV (can pinch and release thumb laterally, fingers can extend in a small semi-voluntary range), V (can do spherical and cylindrical grasp, fingers can extend together but not individually) patients.

[0158] Patient characteristics: The affected hand has only partial voluntary movement, but has relatively reliable electromyographic signals that express movement intention.

[0159] Measures: Assisted movement therapy, collecting electromyographic signals of the affected hand according to specific rehabilitation movements to identify gestures, thereby controlling the rehabilitation assistive device to perform rehabilitation exercises.

[0160] (III) Mildly ill patients

[0161] Applicable subjects: Brunnstorm stage VI patients.

[0162] Patient characteristics: All grasps can be completed, but the speed and accuracy are worse than the healthy side. At this stage, the patient can perform all movements by himself.

[0163] Measures: Assisted movement therapy, collecting electromyographic signals of the affected hand according to specific rehabilitation movements to identify gestures, thereby controlling the rehabilitation assistive device to perform rehabilitation exercises.

[0164] The present invention also provides a memory storing a plurality of instructions, wherein the instructions are used to implement the method described in the first embodiment.

[0165] like Figure 6 As shown, the present invention also provides an electronic device, including a processor 501 and a memory 502 connected to the processor 501, wherein the memory 502 stores a plurality of instructions, which can be loaded and executed by the processor so that the processor can execute the method described in Example 1.

[0166] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A motion intention recognition method based on a dual-stream Transformer encoder and a multi-head attention mechanism, characterized in that: include: S1, obtaining multiple groups of sample information collected by a wearable device worn by a stroke patient, each group of sample information including a sample electromyographic signal, an inertial measurement signal and / or a sample electroencephalographic signal; S2, based on each group of sample information, establishes a motion intention recognition model based on a dual-stream Transformer encoder and a multi-head attention mechanism; S3, determining the movement intention of the stroke patient based on the movement intention recognition model; S2, based on each set of sample information, establishes a motion intention recognition model based on a dual-stream Transformer encoder and a multi-head attention mechanism, including: S21, preprocessing the sample information and obtaining a first part of data set data required for establishing a motion intention recognition model; S22, expanding the first part of the data set to obtain the second part of the data set, and merging the first part of the data set with the second part of the data set to form a sample data set; S23, establishing a motion intention recognition network based on a dual-stream Transformer encoder and a multi-head attention mechanism; the motion intention recognition network includes a dual-stream Transformer encoder, a long-short sequence feature cross attention module, a multi-scale feature fusion module and a motion intention classification module; the dual-stream Transformer encoder includes a multi-head attention mechanism; S24, inputting the sample data set data into the motion intention recognition network for training and learning to obtain the motion intention recognition model; The preprocessing of S21 includes noise reduction, normalization, taking absolute value and data segmentation, wherein: S211, the noise reduction includes filtering out power frequency interference, motion artifacts and / or noise caused by multi-channel crosstalk in the original electromyographic signal, thereby obtaining sample information after the noise is filtered out; S212, the normalization includes limiting the sample information after the noise is filtered out to a suitable size that is conducive to model training, and obtaining a normalized electromyographic signal; S213, taking the absolute value includes: taking the absolute value of all sequences of the normalized electromyographic signals; S214, data segmentation: cutting the entire sequence of normalized electromyographic signals after taking absolute values ​​into a plurality of sample time series windows, and using the plurality of sample time series windows as data set data; The dual-stream Transformer encoder includes a channel attention module, a long sequence slice transformation module, a short sequence slice transformation module, a multi-head attention mechanism module and a feedforward neural network module; The long-short sequence feature cross attention module is used to simultaneously learn the identification information of the long sequence branch and the short sequence branch obtained by the long sequence slice transformation module and the short sequence slice transformation module; The multi-scale feature fusion module is used to fuse the recognition information of the long sequence branch and the recognition information of the short sequence branch through the long-short sequence feature cross attention module and output a multi-scale fusion feature; The motion intention classification module: uses full connection to classify the multi-scale fusion features to obtain a motion intention output result; Building the dual-stream Transformer encoder includes: S231, establishing a channel attention module, including: calculating the connection between each channel according to the temporal characteristics of the sample information and the spatial characteristics of the multi-channel electromyographic signal, learning the importance of the signal characteristics of each channel in the spatial distribution to the rehabilitation movement recognition, and adaptively adjusting the recognition weight of each channel, so that the sample timing window passing through the channel attention module forms the channel attention; S232, establishing a long sequence slicing transformation module and a short sequence slicing transformation module, including: slicing the sample time sequence window forming the channel attention according to the number of sampling points within a certain period of time to form a long sequence with more sampling points and a short sequence with fewer sampling points; transforming the long sequence and the short sequence of each slice into a long sequence slice one-dimensional vector and a short sequence slice one-dimensional vector through a long sequence slicing module and a short sequence slicing module respectively; S233, establish a multi-head attention mechanism module; S234, establishing a feedforward neural network, wherein the feedforward neural network is composed of a plurality of fully connected layers, a first residual connection and a normalization module are set between the feedforward neural network and the multi-head attention mechanism module; a second residual connection and a normalization module are set between the feedforward neural network and the long-short sequence feature cross attention module; The calculation formula of each head of the multi-head attention mechanism in S233 is as shown in formula (1). The calculation formula of the attention mechanism can learn the correlation between each slice in the same layer: ； in Attention(Q,K,V) is a multi-head attention mechanism, Q, K, V are query matrix, key matrix and value matrix respectively, d is the row vector dimension of the matrix, Soft max is the normalized exponential function, Soft The max function is a single-layer neural network and a binary classification function. sigmoid The generalization of multi-classification is used to display the results of multi-classification in the form of probability; The multi-head attention mechanism module includes: A multi-head slice formation module (S1) is used to construct a long sequence slice matrix and a short sequence slice matrix based on the long sequence slice one-dimensional vector and the short sequence slice one-dimensional vector and input them into a slice matrix transformation module (S2); Slice matrix transformation module (S2): used to linearly transform the long sequence slice matrix and the short sequence slice matrix to obtain the query matrix Q、 Key Matrix K Sum Matrix V And obtain the converted query matrix through the fully connected layer Q'、 Key Matrix Sum Matrix ， And input the single-head attention transformation module (S3); Single-head attention transformation module (S3): used to transform the query matrix based on the transformation Q' , key matrix Sum Matrix Get multiple single-head attention matrices; Multi-head attention fusion module (S4): used to concatenate multiple single-head attention matrices to construct a multi-head attention matrix, compress it, and output the compressed multi-head attention matrix X' or Y'; The multi-head slice forming module (S1) is used to receive n short sequence slice one-dimensional vectors a1, a2, ..., a output by the short sequence slice module. n , or receive L long sequence slice one-dimensional vectors b1, b2, ..., b output by the long sequence slice module L ; Construct a short sequence slice matrix X=[a1,a2,…,a n ] or long sequence slice matrix Y=[b1,b2,…,b L ]; outputting the short sequence slice matrix X or the long sequence slice matrix Y to a slice matrix transformation module (S2); The slice matrix transformation module (S2) is used to receive the short sequence slice matrix X or the long sequence slice matrix Y, and obtain the query matrix by linear transformation. Q、 Key Matrix K Sum Matrix V， As shown in formulas (11), (12), and (13); For a short sequence slice matrix X, then: （11）； （12）； （13）； Or perform the same operations as (11), (12), and (13) for the long sequence slice matrix Y; in W q ，W K ，W v ， is a matrix with learnable parameters for each attention mechanism, b q ，b k ，b v It is the matrix bias, which updates the parameter matrix and matrix bias through optimization during model training; Get Q, K, V After the value is obtained, the converted query matrix is ​​obtained through the fully connected layer. Q'， Key Matrix Sum Matrix , the transformed query matrix Q'， Key Matrix Sum Matrix Output to the single-head attention transformation module (S3) to construct a single head of the multi-head attention mechanism; The single-head attention transformation module (S3) is used to receive the transformed query matrix output by the slice matrix transformation module (S2) Q'， Key Matrix Sum Matrix , and then proceed as follows: First, the converted key matrix Transposed and transformed query matrix Q' Dot product; then divide the dot product by the row vector dimension of the matrix d One-half of ; Finally, the above calculation results are Softmax The function normalizes and multiplies the transformed value matrix ， Get the output matrix head containing single-head attention information; The calculation formula of the single-head attention is as follows (1'): ； Where head is the output matrix containing single-head attention information, Attention(Q',K' , ) is a single-head attention transformation, Q', K', are the converted query matrix, key matrix and value matrix respectively, Q', K', The matrices have the same dimensions ，d is the row vector dimension of the matrix; When there are h single-head attention modules, h single-head attention output matrices are obtained respectively. head 1, head 2, …, head i,…,head h , as shown in formula (14): in , 1<=i<=h(14); And output the h single-head attention matrices to the multi-head attention fusion module (S4); The multi-head attention fusion module (S4) is used to receive the h single-head attention output matrices head 1, head 2,…,head i,…,head h , concatenate them to construct a matrix containing multi-head attention information, as described in formula (15), ； The concat function can connect multiple matrices along the specified axis to form a concatenated matrix; The multi-head attention matrix After compression processing is performed to obtain the compressed multi-head attention matrix X' or Y', the compressed multi-head attention matrix X' or Y' is output to the first residual connection and normalization module for processing, wherein the compression processing includes: using a fully connected layer to compress the multi-head attention matrix so that the compressed multi-head attention matrix is ​​consistent with the dimension of the single-head attention matrix.

2. According to claim 1, a motion intention recognition method based on a dual-stream Transformer encoder and a multi-head attention mechanism is characterized in that: The wearable device is an electromyography acquisition sensor, an inertial measurement sensor and / or an electroencephalography acquisition sensor.

3. According to claim 1, a motion intention recognition method based on a dual-stream Transformer encoder and a multi-head attention mechanism is characterized in that: The step S22 of performing data expansion on the first part of the data set to obtain data of the second part of the data set includes: S221, random windowing: randomly windowing the data of the first part of the data set to obtain randomly windowed sample data in the data of the second part of the data set, including: randomly selecting a starting point of a window in each type of action sequence, and determining an end point according to the window length to obtain an electromyographic timing window; based on the electromyographic timing window, randomly windowing the entire sequence of normalized sample information after taking the absolute value to obtain the randomly windowed sample data in the data of the second part of the data set; S222, delay signal enhancement: performing delay signal enhancement on the data of the first part of the data set to obtain the delay signal enhanced sample data in the data of the second part of the data set, including: randomly selecting a section of sampling points of one of the multiple sample timing windows of S214 and deleting it; selecting the sampling points of the next moment of one of the sample timing windows with the same number of deleted sampling points and putting them into the end of the window to form a delay signal enhancement timing window; performing delay signal enhancement on the entire sequence of normalized sample information after taking the absolute value based on the delay signal enhancement timing window to obtain the delay signal enhanced sample data in the data of the second part of the data set; S223, merging the randomly windowed sample data in the second part of the data set and the delayed signal enhanced sample data in the second part of the data set to obtain the second part of the data set; merging the first part of the data set with the second part of the data set to form a sample data set based on data enhancement merging, thereby effectively expanding the data volume of the sample data set.

4. The motion intention recognition method based on dual-stream Transformer encoder and multi-head attention mechanism according to claim 3 is characterized in that: The S22 only includes executing the random windowing in S221 or the delayed signal enhancement in S222, and accordingly chooses not to implement S223.

5. A motion intention recognition system based on a dual-stream Transformer encoder and a multi-head attention mechanism, used to implement the motion intention recognition method described in any one of claims 1 to 4, characterized in that: include: A sample collection module (101) is used to obtain multiple groups of sample information collected by a wearable device worn by a stroke patient, each group of sample information including a sample electromyographic signal, an inertial measurement signal and / or a sample electroencephalographic signal; A model building module (102) is used to build a motion intention recognition model based on a dual-stream Transformer encoder and a multi-head attention mechanism based on each set of sample information; and A movement intention recognition module (103) is used to determine the movement intention of the stroke patient based on the movement intention recognition model.

6. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a plurality of instructions, and the processor is used to read the instructions and execute the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the plurality of instructions can be read by a processor to execute the method according to any one of claims 1 to 4.

Images