Robot interaction-oriented motion intention recognition method, system and equipment and medium
By acquiring electromyography and inertial timing signals and combining the Informer-LSTM model, the problem of multi-joint multi-degree-freedom continuous motion intention recognition is solved, and the accurate identification and prediction of daily life movements is achieved, which improves the continuity and accuracy of recognition .
Patent Information
- Application Number
- CN202411516910.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-29
- Publication Date
- 2024-11-26
AI Technical Summary
The existing multi-degree-of-freedom motion intention recognition technology mainly focuses on the recognition of discrete actions, lacks the ability to recognize movement intentions when switching between multiple joints and multiple degrees of freedom, making it difficult to achieve accurate recognition in daily life actions.
By obtaining the electromyography and inertial timing signals collected synchronously, endpoint recognition is used to decompose the multi-joint continuous motion process, feature extraction and prediction are combined with the Informer-LSTM model, and the sliding window and extended Kalman filtering algorithm are used to improve the recognition accuracy.
Accurate identification of multi-joint continuous motion intentions is achieved, the continuity and accuracy of long-term predictions are improved, and joint rotation and flexion and extension motion intentions can be identified in daily movements.
Smart Images

Figure CN119025900A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of human-machine interaction and artificial intelligence technology, and in particular to a motion intention recognition method, system, device and medium for robot interaction. Background Art
[0002] Studies have shown that in addition to rehabilitation exercises for the affected side to improve the range of motion and functional ability of the joint, adding daily life movement training can improve the patient's performance in daily activities. Daily movement is a complex movement involving multiple joints and multiple degrees of freedom. Recognizing the movement intention of daily movement through electromyographic signals involves the continuous movement intention recognition technology of complex movements with multiple degrees of freedom.
[0003] At present, there are few studies on multi-degree-of-freedom motion intention recognition technology based on electromyographic signals. Common strategies include: 1) Different combinations of actions of different degrees of freedom are defined to form different discrete motion types, and the corresponding surface electromyographic signals are collected to construct a classifier to realize the motion intention recognition between different degrees of freedom action combinations; 2) Another strategy is to construct multiple classifiers according to the number of degrees of freedom. Each classifier processes data in parallel, and the results of each classifier are comprehensively judged through the decision algorithm to obtain the final recognition result; 3) By monitoring the electromyographic signals of the muscles corresponding to the movements of each degree of freedom, when the characteristic amplitude of the target muscle is greater than the established threshold, it is recognized that the movement intention of the human body at this time is the movement of the degree of freedom corresponding to this muscle. For example, when it is detected that the characteristic amplitude of the electromyographic signal of the biceps brachii exceeds the threshold, it is recognized as elbow flexion movement.
[0004] The above methods have achieved good recognition results in 2-3 degrees of freedom movements, proving the feasibility of multi-degree-of-freedom motion recognition. However, current studies all regard each multi-degree-of-freedom movement as a discrete movement, and lack the exploration of intention recognition performance when switching between different movements in continuous motion. Summary of the invention
[0005] In order to achieve the above-mentioned purpose and other advantages of the present invention, the first purpose of the present invention is to provide a motion intention recognition method for robot interaction, comprising the following steps: Acquire synchronously collected timing signals during multi-joint multi-degree-of-freedom continuous motion, including electromyographic timing signals and inertial timing signals; By performing endpoint recognition on the inertial timing signal, the multi-joint multi-degree-of-freedom continuous motion process is decomposed into multiple stages according to each degree of freedom; Segment and extract features of the time series signal at each stage to obtain the time series signal features; The changing trend of the time series signal characteristics is captured based on long-term dependency, and the change of movement intention in a short period of time is solved based on short-term correlation to obtain a continuous movement intention prediction result.
[0006] Furthermore, before performing endpoint recognition on the inertial timing signal to decompose the multi-joint multi-degree-of-freedom continuous motion process into multiple phase steps according to each degree of freedom, the method further includes: Performing bandpass filtering on the original electromyographic timing signal to extract the electromyographic timing signal; Performing band-stop filtering on the extracted electromyographic timing signal to remove power frequency noise in the electromyographic timing signal; The maximum voluntary contraction force signal collected is obtained to perform normalization processing on the electromyographic timing signal.
[0007] Furthermore, the steps of performing endpoint recognition on the inertial timing signal to decompose the multi-joint multi-degree-of-freedom continuous motion process into multiple stages according to each degree of freedom include: A double threshold method is used to identify and segment each motion segment and stationary segment of the inertial time series signal to obtain each endpoint stage; The electromyographic timing signal is decomposed based on each endpoint phase.
[0008] Furthermore, the step of segmenting and extracting features of the time series signal at each stage to obtain time series signal features includes: A sliding window is used to segment the EMG timing signals at each stage; Taking muscle activation as a feature, the average muscle activation value in each sliding window is calculated to obtain the characteristics of the electromyographic timing signal.
[0009] Furthermore, the step of capturing the changing trend of the time series signal characteristics based on long-term dependency and solving the change of movement intention in a short period of time based on short-term correlation to obtain the continuous movement intention prediction result includes: For the electromyographic timing signal characteristics, a continuous movement intention prediction model based on the electromyographic timing signal is used to perform continuous movement intention prediction to obtain a first prediction result; wherein, the continuous movement intention prediction model based on the electromyographic timing signal adopts the Informer-LSTM model.
[0010] Furthermore, the step of predicting the continuous movement intention using a continuous movement intention prediction model based on the electromyographic timing signal for the electromyographic timing signal features includes: The input of the encoder and decoder of the Informer model at the current moment is constructed by extracting the features of the electromyographic timing signal and the output of the Informer model at the previous moment, so as to predict the features of the kth moment in the future. The predicted features of the kth moment in the future and the cell state of the LSTM model at the previous moment And the output of the previous moment As the input of the LSTM model, the prediction result and the cell state at this moment are obtained And the output at this time ; Compare the output of the Informer model at the current moment with the cell state of the LSTM model at the current moment And the output at the current time The next moment of the continuous movement intention prediction model based on the electromyography timing signal is input for prediction.
[0011] Furthermore, the step of segmenting and extracting features of the time series signal at each stage to obtain time series signal features also includes: A sliding window is used to segment the inertial time series signal at each stage; The average value of the inertial time series signal in each sliding window is calculated to obtain the inertial time series signal characteristics.
[0012] Furthermore, the step of capturing the change trend of the time series signal feature based on the long-term dependency and solving the change of the motion intention in a short time based on the short-term correlation to obtain the continuous motion intention prediction result also includes: For the inertial time series signal feature, a continuous movement intention prediction model based on the inertial time series signal is used to predict the continuous movement intention to obtain a second prediction result; wherein the continuous movement intention prediction model based on the inertial time series signal adopts an LSTM model; The extended Kalman filter algorithm is used to fuse and calibrate the first prediction result and the second prediction result to obtain a continuous motion intention prediction result.
[0013] Furthermore, the decomposed stages include initial position rest, joint supination, supination posture maintenance, joint flexion, flexion posture maintenance, joint extension, extension posture maintenance, and joint pronation.
[0014] Furthermore, capturing the change trend of the electromyographic timing signal characteristics based on long-term dependency and solving the change of movement intention in a short period of time based on short-term correlation also includes: The continuous movement intention prediction model based on the electromyography timing signal and the continuous movement intention prediction model based on the inertia timing signal are used to predict the joint rotation movement intention; When it is predicted that the supination posture is maintained, the continuous movement intention prediction model based on the electromyographic timing signal and the continuous movement intention prediction model based on the inertial timing signal are used to perform joint flexion and extension movement intention recognition; When it is predicted that the joint extension movement is finished, the continuous movement intention prediction model based on the electromyographic timing signal and the continuous movement intention prediction model based on the inertial timing signal are returned to perform joint rotation movement intention prediction and continue execution.
[0015] The second purpose of the present invention is to provide a motion intention recognition system for robot interaction, which is used to implement the above-mentioned method, including a surface electromyography testing system and a processor. The surface electromyography testing system is used to synchronously collect timing signals during multi-joint multi-degree-of-freedom continuous motion, including electromyography timing signals and inertial timing signals; the processor is used to process the electromyography timing signals and the inertial timing signals.
[0016] The third object of the present invention is to provide an electronic device, comprising: a memory on which a program code is stored; a processor connected to the memory and implementing the above method when the program code is executed by the processor.
[0017] A fourth object of the present invention is to provide a computer-readable storage medium having program instructions stored thereon, wherein the program instructions implement the above method when executed by a processor.
[0018] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a motion intention recognition method, system, device and medium for robot interaction, the method comprising the following steps: obtaining synchronously collected timing signals in a multi-joint multi-degree-of-freedom continuous motion process, including electromyographic timing signals and inertial timing signals; performing endpoint recognition on the inertial timing signals to decompose the multi-joint multi-degree-of-freedom continuous motion process into multiple stages according to each degree of freedom; segmenting and feature extracting the timing signals of each stage to obtain timing signal features; capturing the changing trend of the timing signal features based on long-term dependency and solving the motion intention changes in a short period of time based on short-term correlation to obtain continuous motion intention prediction results. The present invention can realize the recognition of multi-joint linkage continuous motion intentions for daily motion actions, and improves the continuity and accuracy of prediction results during long-term predictions based on long-term dependency and short-term correlation.
[0019] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. The specific implementation of the present invention is given in detail by the following embodiments and their accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 The process of the motion intention recognition method for robot interaction of embodiment 1 is as follows Figure 1 ; Figure 2 The process of the motion intention recognition method for robot interaction of embodiment 1 is as follows Figure 2 ; Figure 3 The figure is a flow chart of myoelectric timing signal preprocessing in Example 1; Figure 4 This is a decomposition flow chart of multi-joint multi-degree-of-freedom continuous motion in Example 1; Figure 5 The flowchart of myoelectric timing signal feature extraction of Example 1; Figure 6 This is a flow chart of inertial timing signal feature extraction in Example 1; Figure 7 This is a continuous motion intention prediction flow chart of Example 1; Figure 8 The flowchart of continuous movement intention prediction based on electromyographic timing signal of Example 1; Fig. 9 It is a flow chart of joint pronation / supination intention prediction and joint flexion / extension intention prediction of Example 1; Fig.10 This is a schematic diagram of the Informer-LSTM model of Example 1; Fig.11 Schematic diagram of the Informer model of Example 1 Figure 1 ; Fig.12 Schematic diagram of the Informer model of Example 1 Figure 2 ; Fig.13 This is a schematic diagram of the LSTM model of Example 1; Fig.14 This is a schematic diagram of the eating process of Example 1; Fig.15 This is a schematic diagram of the result of data decomposition based on the inertial time series signal in Example 1; Fig.16 This is a schematic diagram of the prediction of movement intention during eating in Example 1; Fig.17 is a schematic diagram of an electronic device of Example 3; Fig.18 This is a schematic diagram of the storage medium of Example 4. DETAILED DESCRIPTION
[0021] The present invention is further described below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. It should be noted that, under the premise of no conflict, the embodiments or technical features described below can be arbitrarily combined to form a new embodiment.
[0022] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.
[0023] The figure numbers in this application are only used to distinguish the various steps in the scheme and are not used to limit the execution order of the various steps. The specific execution order is subject to the description in the specification.
[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. Example 1
[0025] A motion intention recognition method for robot interaction, such as Figure 1 , Figure 2 As shown, the following steps are included: S1, obtaining synchronously collected timing signals during multi-joint multi-degree-of-freedom continuous motion, including electromyographic timing signals (EMG signals) and inertial timing signals (IMU signals); In the process of multi-degree-of-freedom continuous motion, the different movements are related in terms of timing. The timing of the actions in the process of multi-degree-of-freedom continuous motion can be used, and then combined with the model of timing signals to identify motion intentions, which can improve the continuity and accuracy of the prediction results.
[0026] Furthermore, if Figure 3 As shown, before performing endpoint recognition on the inertial timing signal to decompose the multi-joint multi-degree-of-freedom continuous motion process into multiple phase steps according to each degree of freedom, it also includes: S100, performing bandpass filtering on the original electromyographic timing signal to extract the electromyographic timing signal; S110, performing band-stop filtering on the extracted electromyographic timing signal to remove power frequency noise in the electromyographic timing signal; For example, a 4th-order Butterworth bandpass filter with a frequency range of 30 to 500 Hz is used to process the original EMG data to extract the EMG timing signal; then, a 4th-order Butterworth bandstop filter with a frequency range of 48 to 52 Hz is used to remove the power frequency noise in the EMG timing signal.
[0027] S120, obtaining the collected maximum voluntary contraction force (mvc) signal, performing normalization processing on the electromyographic timing signal, and converting the electromyographic timing signal data into a dimensionless signal of [-1, 1].
[0028] S2, performing endpoint recognition on the inertial timing signal to decompose the multi-joint multi-degree-of-freedom continuous motion process into multiple stages according to each degree of freedom; In some embodiments, Figure 4 As shown, the steps of performing endpoint recognition on the inertial timing signal to decompose the multi-joint multi-degree-of-freedom continuous motion process into multiple stages according to each degree of freedom include: S21, using a double threshold method to identify and segment each motion segment and stationary segment of the inertial time series signal to obtain each endpoint stage; Because the inertial timing signal is more capable of reflecting the movement and static state of the joint than the electromyographic timing signal, the deviation of the decomposition of the multi-joint multi-degree-of-freedom continuous motion process is avoided. For example, when the patient completes the supination of a joint and is in a posture holding state, the patient's muscles are still continuously exerting force, but the posture signal has not changed. Although the electromyographic signal can be detected at this time, the patient has not made further movements. If the motion process is decomposed based on the electromyographic signal, this posture holding state will also be decomposed into the motion stage, causing a deviation in the prediction of the motion intention. Based on this, the present embodiment decomposes the multi-joint multi-degree-of-freedom continuous motion process based on the inertial timing signal, avoids decomposing the posture holding state into the motion stage, and can improve the accuracy of the motion intention prediction.
[0029] S22, decomposing the electromyographic timing signal based on each endpoint stage.
[0030] Specifically, the endpoints identified by the IMU signal are used as the endpoints of the EMG signal, and the entire movement process is decomposed into n stages.
[0031] S3, segmenting and extracting features of the time series signal at each stage to obtain time series signal features; In some embodiments, Figure 5 As shown, the steps of segmenting and extracting features of the time series signal at each stage to obtain the time series signal features include: S31, using a sliding window to perform data segmentation on the electromyographic timing signal of each stage; For example, in order to have better performance in real-time classification, the EMG timing signal is segmented using a sliding window. Since the maximum delay that the human body can perceive is 300ms, a sliding window with a length of 300ms and a step size of 20ms can be selected for data segmentation.
[0032] S32. Using muscle activation as a feature, calculate the average muscle activation value in each sliding window to obtain the electromyography timing signal feature.
[0033] S4. Capturing the changing trend of the time series signal characteristics based on long-term dependency and solving the change of motion intention in a short period of time based on short-term correlation to obtain a continuous motion intention prediction result.
[0034] The above steps are used to decompose a daily movement action into multiple stages according to each degree of freedom, and then a movement intention prediction model can be constructed based on each degree of freedom.
[0035] In some embodiments, Figure 7 As shown, the steps of capturing the changing trend of the time series signal features based on long-term dependency and solving the change of motion intention in a short period of time based on short-term correlation to obtain the continuous motion intention prediction result include: S410. For the electromyographic timing signal features, a continuous movement intention prediction model based on the electromyographic timing signal is used to predict the continuous movement intention to obtain a first prediction result; wherein, the continuous movement intention prediction model based on the electromyographic timing signal adopts an Informer-LSTM model.
[0036] The overall architecture of the Informer-LSTM model is as follows Fig.10 As shown, it is different from the existing models. In the related technology, the Informer model and the LSTM model are used to extract features from data, or the LSTM model and the Informer model are trained separately to obtain two separate models. First, a sequence signal feature is input into the LSTM model for feature extraction, and the obtained new features are combined with the features of the original signal as the input of the Informer for prediction. This method first needs to input a period of sequence data into the LSTM model. If the data feature dimension of a moment is n-dimensional, and there are m moments in total, then the input dimension is For LSTM model, the LSTM model will output a time series feature, which will be combined with the original time series feature and input into the Informer model. When the feature dimension to be input is large or the time dimension is long, this method will result in a huge amount of input data and calculated data, which will lead to a delay increase in a prediction cycle, making it difficult to achieve online motion intention recognition.
[0037] The structure of the Informer model is as follows Fig.11 As shown in Figure 1, it consists of an encoder and a decoder. The Informer model is an improvement on the Transformer model. It proposes a probabilistic sparse self-attention mechanism, which effectively replaces the traditional self-attention mechanism and reduces the computational complexity. , and They are Query vector, Key vector and Value vector respectively, and and express , and No. Row, No. The attention of the query vector of a row is defined as a kernel smoother in probabilistic form: ; in, is the attention probability distribution, is a function that selects the asymmetric exponential kernel . Using KL divergence to detect attention probability distribution With uniform distribution The overlap between them is used to filter out the ones with large changes : ; Delete the constant term and simplify the formula, taking the difference between the maximum value of the probability distribution and the uniform distribution as the output. The simplified formula is: ; Using the above formula to reduce the complexity of the attention mechanism, the final formula of the probabilistic sparse self-attention mechanism is: ; The Informer model introduces distillation operations at the encoder layer, and the computational complexity can be further reduced by half at each distillation layer, thereby solving the problem that the model cannot be stacked due to too long input. Fig.12 shown.
[0038] The Informer decoder uses generative prediction to directly output multi-step predictions. With a single forward inference, this method can predict all outputs of a long sequence.
[0039] The LSTM model adds a gating mechanism based on the RNN model to solve the problem of poor long-term dependence of RNN, which leads to gradient disappearance. The gating mechanism consists of three gates: forget gate, update gate, and output gate. The forget gate determines the information lost by the cell state, the update gate determines the cell state that stores new information, and the output gate determines the information to be output. Fig.13 shows the structure of the LSTM model, is the cell state at time t, is the cell state at time t-1, represents the forget gate, represents the input gate, represents the output gate, and the mathematical formula is: ; in, express function, is the hyperbolic tangent function, , , , are the hidden layer weights of the state vector, , , , is the bias vector on the corresponding gate, is the output at the previous moment, The input at the current moment.
[0040] In some embodiments, Figure 8 As shown, the step of predicting continuous movement intention using a continuous movement intention prediction model based on the electromyography timing signal for the electromyography timing signal features includes: In order to reduce the amount of data calculated and the cycle of motion intention recognition, S411, the input of the encoder and decoder of the Informer model at the current moment is constructed by extracting the characteristics of the electromyographic timing signal and the output of the Informer model at the previous moment, so as to predict the characteristics of the kth moment in the future; S412: The predicted feature of the kth moment in the future is used as the input of a cell in the LSTM model, and the cell state of the previous moment in the LSTM model is used as the input of a cell in the LSTM model. And the output of the previous moment As the input of the LSTM model, the prediction result and the cell state at this moment are obtained And the output at this time , the cell state at this moment is obtained And the output at this time Transfer to the next moment, so as to realize the LSTM model to transfer the short-term features of the time series signal; S413, compare the output of the Informer model at the current moment with the cell state of the LSTM model at the current moment And the output at the current time The next moment of the continuous movement intention prediction model based on the electromyography timing signal is input for prediction.
[0041] This embodiment innovatively proposes an Informer-LSTM model, which can improve the accuracy of continuous motion intention recognition; in the Informer-LSTM model provided in this embodiment, the LSTM model only needs to process an m-dimensional feature at each moment, and the amount of data that needs to be processed at each moment is greatly reduced, thereby reducing the calculation time.
[0042] In order to further improve the accuracy of the prediction, in some embodiments, such as Figure 6 As shown, the step of segmenting and extracting features of the time series signal at each stage to obtain the time series signal features also includes: S33, using a sliding window to perform data segmentation on the inertial time series signal of each stage; the details can refer to the above-mentioned electromyography time series signal segmentation scheme, the principle is the same, and it will not be repeated here.
[0043] S34, calculating the average value of the inertial time series signal in each sliding window to obtain the inertial time series signal characteristics.
[0044] In some embodiments, Figure 7 As shown, the step of capturing the change trend of the time series signal characteristics based on long-term dependency and solving the change of movement intention in a short period of time based on short-term correlation to obtain the continuous movement intention prediction result also includes: S420, for the inertial time series signal feature, using a continuous movement intention prediction model based on the inertial time series signal to predict the continuous movement intention, and obtain a second prediction result; wherein the continuous movement intention prediction model based on the inertial time series signal adopts an LSTM model; S430: Using an extended Kalman filter algorithm to fuse and calibrate the first prediction result and the second prediction result to obtain a continuous motion intention prediction result.
[0045] This embodiment uses the LSTM model to build a continuous motion intention prediction model based on the IMU signal, and uses the extended Kalman filter to fuse the prediction results of the EMG signal and the IMU signal, further improving the accuracy of the prediction.
[0046] In some embodiments, the multi-joint multi-degree-of-freedom continuous motion process is decomposed according to each degree of freedom to obtain the stages including initial position rest, joint supination, supination posture maintenance, joint flexion, flexion posture maintenance, joint extension, extension posture maintenance, and joint pronation. This embodiment adds recognition of posture maintenance status to achieve more coherent continuous motion intention recognition.
[0047] In some embodiments, Fig. 9 As shown, the method of capturing the changing trend of the electromyographic timing signal characteristics based on long-term dependency and solving the change of movement intention in a short period of time based on short-term correlation also includes: S400, using the continuous movement intention prediction model based on the electromyography timing signal and the continuous movement intention prediction model based on the inertia timing signal to predict the joint rotation movement intention; S401, when it is predicted that the supination posture is maintained, the continuous movement intention prediction model based on the electromyographic timing signal and the continuous movement intention prediction model based on the inertial timing signal are used to perform joint flexion and extension movement intention recognition; S402. When it is predicted that the joint extension movement is finished, return to the continuous movement intention prediction model based on the electromyography timing signal and the continuous movement intention prediction model based on the inertia timing signal to perform joint rotation movement intention prediction and continue execution.
[0048] Adding the recognition of isometric contraction postures of muscles moving in each degree of freedom to the continuous motion intention prediction scheme for multi-joint linkage in daily movement actions can improve the consistency of the continuous motion intention prediction results.
[0049] In this embodiment, the principles of the continuous motion intention prediction model based on electromyography timing signals and the continuous motion intention prediction model based on inertial timing signals for predicting joint rotation motion intention and identifying joint flexion and extension motion intention are the same as the principles of the corresponding models in the above embodiments, and can be referred to the above embodiments and will not be repeated here.
[0050] This embodiment builds a continuous motion intention prediction strategy for daily motion based on the proposed Informer-LSTM model, which is used to decode EMG signals and IMU signals in multi-joint multi-degree-of-freedom continuous tasks, and increase the recognition of posture holding status to achieve more coherent continuous motion intention recognition. The Informer model is combined with the LSTM model to construct a continuous motion intention prediction model based on surface electromyography timing signals, and the long-term dependency of the Informer model is used to capture the changing trend of the electromyography timing signal characteristics, and the short-term correlation of the LSTM model is used to solve the change of motion intention in a short time, so as to improve the continuity and accuracy of the prediction results during long-term prediction. On this basis, the LSTM model is used to construct a continuous motion intention prediction model based on IMU signals, and the prediction results of EMG signals and IMU signals are fused using the extended Kalman filter to further improve the accuracy of the prediction.
[0051] In some embodiments, a surface electromyography test system is used to synchronously collect EMG signals and IMU signals, and the EMG signal is a sEMG signal. The sampling frequency of the sEMG signal is 2148 Hz, and the sampling frequency of the IMU signal is 370 Hz. Before placing the electrodes on the skin, wipe the skin with 70% alcohol to remove impurities on the epidermis.
[0052] The eating action is used as an example for illustration, and should not be understood as a restriction on the types of multi-joint multi-DOF continuous motion actions. Eating action: The subject is tested in a sitting position, with a table in front of the subject and a bowl of water on the table. The subject holds the spoon in reverse with one end of the spoon closest to the little finger and the other end closest to the thumb.
[0053] The initial position is: the subject places the elbow on the table, the supinator and pronator teres are in a relaxed state, the biceps are in an isometric contraction state, and the spoon is placed in the bowl in front of the face, and the spoon is in the water. After the experiment begins, the subject completes the first complete action according to the following steps. Each subject needs to complete 10 complete actions in each experiment. Each subject conducts 5 data collection experiments, and rests for 5 minutes between each experiment to prevent fatigue.
[0054] After the experiment started, the subjects performed supination and extension movements at the elbow joint, scooped water with a spoon in their hand, and maintained the posture for 5 seconds after the movement was completed; Flex the elbow joint while maintaining a supinated position, so that the spoon can be moved to the mouth without spilling the water in the spoon. After the movement is completed, maintain the position for 5 seconds. Extend your elbows while maintaining a supinated position, moving the spoon back over the bowl, and hold the position for 5 seconds. Pronate your elbow until you put the spoon back into the bowl and return to the initial position. After completing the movement, maintain the posture for 10 seconds before starting the next movement.
[0055] The electromyographic timing signals (EMG signals) and inertial timing signals (IMU signals) synchronously collected during the execution of the above-mentioned eating action are processed, including digital filtering, normalization processing, endpoint recognition, data segmentation, feature extraction, etc. The specific processing process is the same as the corresponding scheme in the above-mentioned embodiment, and will not be repeated here.
[0056] Taking the decomposition of eating action as an example, the endpoints identified by IMU signals are used as the endpoints of sEMG signals. In the eating action, the entire movement process is identified according to the two states of motion and stillness. The entire eating process can be divided into n stages, such as Fig.14 As shown; and the entire eating process is divided into two parts according to the pronation / supination freedom and the flexion / extension freedom, as shown Fig.15 shown.
[0057] According to the decomposition results of the eating action, the entire recognition and classification process is divided into two stages: pronation / supination intention prediction and flexion / extension intention prediction. The recognition order is as follows: Fig.16As shown in the figure, the first stage of intention prediction is pronation / supination intention prediction. When the supination posture is recognized, the second stage of intention prediction begins. When the stretching action is predicted to be completed, the intention prediction of the first stage is returned. The intention prediction of each stage adopts a continuous motion intention prediction model based on electromyographic timing signals and a continuous motion intention prediction model based on inertial timing signals. The continuous motion intention prediction model based on electromyographic timing signals adopts the above-mentioned Informer-LSTM model, and the continuous motion intention prediction model based on inertial timing signals adopts the LSTM model. The intention prediction results of the two models are fused and calibrated using the extended Kalman filter to obtain more accurate motion intention prediction results.
[0058] In some embodiments, after predicting the human body's movement intention, the exoskeleton robot can be controlled to provide assistance to assist the patient in daily life movement training to improve the patient's performance in daily activities.
[0059] This embodiment provides a method for human motion intention recognition for robot human-machine interaction, which decomposes multi-joint multi-degree-of-freedom continuous motion into multiple stages, establishes continuous motion intention prediction models based on electromyographic signals and posture signals, and recognizes the intention to switch motion during multi-joint multi-degree-of-freedom continuous motion in 3D space, which is used to control the rehabilitation robot to move in different directions. At the same time, for the prediction of continuous motion intention of daily movement movements, in addition to flexion and extension (isometric contraction), the movement of a single degree of freedom also has a posture holding state, that is, a muscle isometric contraction state. Unlike the resting state, the muscle remains in a contracted state in the isometric contraction holding state, so the electromyographic signal can be continuously detected. This embodiment adds the recognition of muscle isometric contraction intention, solves the problem of identifying the human body's muscle isometric contraction state as flexion or extension in the related technology, and improves the final continuous motion intention recognition performance. Example 2
[0060] A motion intention recognition system for robot interaction is used to implement the above method. For a detailed description of the method, reference can be made to the corresponding description in the above method embodiment, which will not be repeated here.
[0061] The present embodiment provides a motion intention recognition system for robot interaction, including a surface electromyography testing system and a processor. The surface electromyography testing system is used to synchronously collect timing signals during multi-joint multi-degree-of-freedom continuous motion, including electromyography timing signals and inertial timing signals; the processor is used to process the electromyography timing signals and the inertial timing signals.
[0062] The motion intention recognition system for robot interaction of this embodiment may include the following process: A synchronous timing signal acquisition module is used to acquire synchronously collected timing signals during multi-joint multi-degree-of-freedom continuous motion, including electromyographic timing signals and inertial timing signals; A continuous motion process decomposition module is used to decompose the multi-joint multi-degree-of-freedom continuous motion process into multiple stages according to each degree of freedom by performing endpoint recognition on the inertial timing signal; The time series signal feature extraction module is used to segment and extract features of the time series signal at each stage to obtain the time series signal features; The continuous motion intention prediction module is used to capture the change trend of the time series signal characteristics based on long-term dependency and solve the change of motion intention in a short period of time based on short-term correlation to obtain a continuous motion intention prediction result.
[0063] Based on the technical solution of the above embodiment, optionally, before performing endpoint recognition on the inertial timing signal to decompose the multi-joint multi-degree-of-freedom continuous motion process into multiple stage steps according to each degree of freedom, the method further includes: Performing bandpass filtering on the original electromyographic timing signal to extract the electromyographic timing signal; Performing band-stop filtering on the extracted electromyographic timing signal to remove power frequency noise in the electromyographic timing signal; The maximum voluntary contraction force signal collected is obtained to perform normalization processing on the electromyographic timing signal.
[0064] Based on the technical solution of the above embodiment, optionally, the step of performing endpoint recognition on the inertial timing signal to decompose the multi-joint multi-degree-of-freedom continuous motion process into multiple stages according to each degree of freedom includes: A double threshold method is used to identify and segment each motion segment and stationary segment of the inertial time series signal to obtain each endpoint stage; The electromyographic timing signal is decomposed based on each endpoint phase.
[0065] Based on the technical solution of the above embodiment, optionally, the step of segmenting and extracting features of the time series signal of each stage to obtain the time series signal features includes: A sliding window is used to segment the EMG timing signals at each stage; Taking muscle activation as a feature, the average muscle activation value in each sliding window is calculated to obtain the characteristics of the electromyographic timing signal.
[0066] Based on the technical solution of the above embodiment, optionally, the step of capturing the change trend of the time series signal characteristics based on long-term dependency and solving the change of movement intention in a short period of time based on short-term correlation to obtain the continuous movement intention prediction result includes: For the electromyographic timing signal characteristics, a continuous movement intention prediction model based on the electromyographic timing signal is used to perform continuous movement intention prediction to obtain a first prediction result; wherein, the continuous movement intention prediction model based on the electromyographic timing signal adopts the Informer-LSTM model.
[0067] Based on the technical solution of the above embodiment, optionally, the step of predicting the continuous movement intention by using a continuous movement intention prediction model based on the electromyography timing signal for the electromyography timing signal feature includes: The input of the encoder and decoder of the Informer model at the current moment is constructed by extracting the features of the electromyographic timing signal and the output of the Informer model at the previous moment, so as to predict the features of the kth moment in the future. The predicted features of the kth moment in the future and the cell state of the LSTM model at the previous moment And the output of the previous moment As the input of the LSTM model, the prediction result and the cell state at this moment are obtained And the output at this time ; Compare the output of the Informer model at the current moment with the cell state of the LSTM model at the current moment And the output at the current time The next moment of the continuous movement intention prediction model based on the electromyography timing signal is input for prediction.
[0068] Based on the technical solution of the above embodiment, optionally, the step of segmenting and extracting features of the time series signal of each stage to obtain time series signal features further includes: A sliding window is used to segment the inertial time series signal at each stage; The average value of the inertial time series signal in each sliding window is calculated to obtain the inertial time series signal characteristics.
[0069] Based on the technical solution of the above embodiment, optionally, the step of capturing the change trend of the time series signal characteristics based on long-term dependency and solving the change of movement intention in a short period of time based on short-term correlation to obtain the continuous movement intention prediction result also includes: For the inertial time series signal feature, a continuous movement intention prediction model based on the inertial time series signal is used to predict the continuous movement intention to obtain a second prediction result; wherein the continuous movement intention prediction model based on the inertial time series signal adopts an LSTM model; The extended Kalman filter algorithm is used to fuse and calibrate the first prediction result and the second prediction result to obtain a continuous motion intention prediction result.
[0070] Based on the technical solution of the above embodiment, optionally, the decomposed stages include resting in the initial position, joint supination, maintaining the supination posture, joint flexion, maintaining the flexion posture, joint extension, maintaining the extension posture, and joint pronation.
[0071] Based on the technical solution of the above embodiment, optionally, capturing the change trend of the electromyographic timing signal characteristics based on long-term dependency and solving the change of movement intention in a short period of time based on short-term correlation further includes: The continuous movement intention prediction model based on the electromyography timing signal and the continuous movement intention prediction model based on the inertia timing signal are used to predict the joint rotation movement intention; When it is predicted that the supination posture is maintained, the continuous movement intention prediction model based on the electromyographic timing signal and the continuous movement intention prediction model based on the inertial timing signal are used to perform joint flexion and extension movement intention recognition; When it is predicted that the joint extension movement is finished, the continuous movement intention prediction model based on the electromyographic timing signal and the continuous movement intention prediction model based on the inertial timing signal are returned to perform joint rotation movement intention prediction and continue execution.
[0072] This embodiment provides a human motion intention recognition system for robot human-machine interaction, which decomposes multi-joint multi-degree-of-freedom continuous motion into multiple stages, establishes continuous motion intention prediction models based on electromyographic signals and posture signals, and recognizes the switching motion intention during multi-joint multi-degree-of-freedom continuous motion in 3D space, which is used to control the rehabilitation robot to move in different directions. At the same time, for the prediction of continuous motion intention of daily movement movements, in addition to flexion and extension (isometric contraction), the movement of a single degree of freedom also has a posture holding state, that is, a muscle isometric contraction state. Unlike the resting state, the muscle remains in a contracted state in the isometric contraction holding state, so the electromyographic signal can be continuously detected. This embodiment adds the recognition of muscle isometric contraction intention, solves the problem of identifying the human body's muscle isometric contraction state as flexion or extension in the related technology, and improves the final continuous motion intention recognition performance. Example 3
[0073] An electronic device, such as Fig.17 As shown, it includes: a memory on which program codes are stored; a processor connected to the memory, and when the program codes are executed by the processor, a motion intention recognition method for robot interaction is implemented. For a detailed description of the method, reference can be made to the corresponding description in the above method embodiment, which will not be repeated here. Example 4
[0074] A computer readable storage medium such as Fig.18As shown, program instructions are stored thereon, and when the program instructions are executed by the processor, a motion intention recognition method for robot interaction is implemented. For a detailed description of the method, reference may be made to the corresponding description in the above method embodiment, and no further description is given here.
[0075] The number of devices and processing scales described here are used to simplify the description of the present invention. Applications, modifications and variations of the present invention will be obvious to those skilled in the art.
[0076] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and implementation modes. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.
[0077] The apparatus, electronic device, non-volatile computer storage medium and method provided in the embodiments of this specification correspond to each other, and therefore, the apparatus, electronic device and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, electronic device and non-volatile computer storage medium will not be repeated here.
[0078] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software units for implementing the method and structures within the hardware component.
[0079] The systems, devices or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described separately by functions in various units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in the same or more software and / or hardware.
[0080] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0081] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0082] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0083] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0084] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0085] The specification may be described in the general context of computer-executable instructions executed by a computer, such as program units. Generally, program units include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program units may be located in local and remote computer storage media, including storage devices.
[0086] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0087] The above description is only an embodiment of this specification and is not intended to limit one or more embodiments of this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of one or more embodiments of this specification.
Claims
1. A motion intention recognition method for robot interaction, characterized in that: The following steps are involved: Acquire synchronously collected timing signals during multi-joint multi-degree-of-freedom continuous motion, including electromyographic timing signals and inertial timing signals; By performing endpoint recognition on the inertial timing signal, the multi-joint multi-degree-of-freedom continuous motion process is decomposed into multiple stages according to each degree of freedom; Segment and extract features of the time series signal at each stage to obtain the time series signal features; The changing trend of the time series signal characteristics is captured based on long-term dependency, and the change of movement intention in a short period of time is solved based on short-term correlation to obtain a continuous movement intention prediction result.
2. A method for identifying motion intentions for robot interaction as claimed in claim 1, characterized in that: Before performing endpoint recognition on the inertial timing signal to decompose the multi-joint multi-degree-of-freedom continuous motion process into multiple phase steps according to each degree of freedom, the method further includes: Performing bandpass filtering on the original electromyographic timing signal to extract the electromyographic timing signal; Performing band-stop filtering on the extracted electromyographic timing signal to remove power frequency noise in the electromyographic timing signal; The maximum voluntary contraction force signal collected is obtained to perform normalization processing on the electromyographic timing signal.
3. A method for identifying motion intentions for robot interaction as claimed in claim 1, characterized in that: The steps of performing endpoint recognition on the inertial timing signal to decompose the multi-joint multi-degree-of-freedom continuous motion process into multiple stages according to each degree of freedom include: A double threshold method is used to identify and segment each motion segment and stationary segment of the inertial time series signal to obtain each endpoint stage; The electromyographic timing signal is decomposed based on each endpoint phase.
4. A method for identifying motion intentions for robot interaction as claimed in claim 3, characterized in that: The step of segmenting and extracting features of the time series signal at each stage to obtain the features of the time series signal includes: A sliding window is used to segment the EMG timing signals at each stage; Taking muscle activation as a feature, the average muscle activation value in each sliding window is calculated to obtain the characteristics of the electromyographic timing signal.
5. A method for identifying motion intentions for robot interaction as claimed in claim 4, characterized in that: The step of capturing the change trend of the time series signal feature based on the long-term dependency and solving the change of the motion intention in a short period of time based on the short-term correlation to obtain the continuous motion intention prediction result includes: For the electromyographic timing signal characteristics, a continuous movement intention prediction model based on the electromyographic timing signal is used to perform continuous movement intention prediction to obtain a first prediction result; wherein, the continuous movement intention prediction model based on the electromyographic timing signal adopts the Informer-LSTM model.
6. A method for identifying motion intentions for robot interaction as claimed in claim 5, characterized in that: The step of predicting the continuous movement intention using a continuous movement intention prediction model based on the electromyographic timing signal for the electromyographic timing signal features comprises: The input of the encoder and decoder of the Informer model at the current moment is constructed by extracting the features of the electromyographic timing signal and the output of the Informer model at the previous moment, so as to predict the features of the kth moment in the future. The predicted features of the kth moment in the future and the cell state of the LSTM model at the previous moment And the output of the previous moment As the input of the LSTM model, the prediction result and the cell state at this moment are obtained And the output at this time ; Compare the output of the Informer model at the current moment with the cell state of the LSTM model at the current moment And the output at the current time The next moment of the continuous movement intention prediction model based on the electromyography timing signal is input for prediction.
7. A method for identifying motion intentions for robot interaction as claimed in claim 5, characterized in that: The step of segmenting and extracting features of the time series signal at each stage to obtain the features of the time series signal also includes: A sliding window is used to segment the inertial time series signal at each stage; The average value of the inertial time series signal in each sliding window is calculated to obtain the inertial time series signal characteristics.
8. A method for identifying motion intentions for robot interaction as claimed in claim 7, characterized in that: The step of capturing the change trend of the time series signal feature based on the long-term dependency and solving the change of the movement intention in a short period of time based on the short-term correlation to obtain the continuous movement intention prediction result also includes: For the inertial time series signal feature, a continuous movement intention prediction model based on the inertial time series signal is used to predict the continuous movement intention to obtain a second prediction result; wherein the continuous movement intention prediction model based on the inertial time series signal adopts an LSTM model; The extended Kalman filter algorithm is used to fuse and calibrate the first prediction result and the second prediction result to obtain a continuous motion intention prediction result.
9. A method for identifying motion intentions for robot interaction according to claim 8, characterized in that: The decomposed stages include initial resting position, joint supination, supination posture maintenance, joint flexion, flexion posture maintenance, joint extension, extension posture maintenance, and joint pronation.
10. A method for identifying motion intentions for robot interaction according to claim 9, characterized in that: The method of capturing the change trend of the electromyographic time series signal characteristics based on long-term dependency and solving the change of movement intention in a short period of time based on short-term correlation also includes: The continuous movement intention prediction model based on the electromyography timing signal and the continuous movement intention prediction model based on the inertia timing signal are used to predict the joint rotation movement intention; When it is predicted that the supination posture is maintained, the continuous movement intention prediction model based on the electromyographic timing signal and the continuous movement intention prediction model based on the inertial timing signal are used to perform joint flexion and extension movement intention recognition; When it is predicted that the joint extension movement is finished, the continuous movement intention prediction model based on the electromyographic timing signal and the continuous movement intention prediction model based on the inertial timing signal are returned to perform joint rotation movement intention prediction and continue execution.
11. A motion intention recognition system for robot interaction, used to implement the method according to any one of claims 1 to 10, characterized in that: It comprises a surface electromyography test system and a processor. The surface electromyography test system is used to synchronously collect timing signals in a multi-joint multi-degree-of-freedom continuous motion process, including electromyography timing signals and inertia timing signals; and the processor is used to process the electromyography timing signals and the inertia timing signals.
12. An electronic device, characterized in that: include: A memory having program code stored therein; A processor connected to the memory, and when the program code is executed by the processor, the method according to any one of claims 1 to 10 is implemented.
13. A computer-readable storage medium, characterized in that: Program instructions are stored thereon, and when the program instructions are executed by a processor, the method according to any one of claims 1 to 10 is implemented.
Citation Information
Patent Citations
Cross-user gesture recognition method based on multi-source domain self-adaption
CN115512440A
