Mechanical arm control method based on neural decoding and body intelligence

Abstract

The invention provides a mechanical arm control method based on neural decoding and body intelligence, which comprises the following steps of: collecting electromyographic signals of limbs of a user, and filtering the electromyographic signals; calculating time domain features among different channels of the electromyographic signals and reconstructing frequency domain features among the different channels; performing feature fusion processing and analysis on the time domain feature and the frequency domain feature through a neural network to obtain a user intention; information of the environment where the mechanical arm is located is collected through a sensor; the motion track of the mechanical arm is planned according to the environment information and the user intention through the body model; according to the scheme, the accuracy of the control instruction analysis result can be improved, the motion smoothness / flexibility of the mechanical arm is improved, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of human-machine environment interaction, and in particular to a robotic arm control method based on neural decoding and embodied intelligence. Background Art

[0002] With the development of technology, human-machine-environment interaction has gradually become a research hotspot in the fields of artificial intelligence and robotics, and has great value in scenarios such as smart homes, rehabilitation medicine, virtual reality, and smart industries. Human-machine-environment interaction refers to the process of interaction between humans, mechanical systems, and the environment. In this interaction, the environment provides the place and background for interactive activities. Humans exchange information with mechanical systems through various input and output devices, such as keyboards and mice, touch screens, voice recognizers, and electromyographic signal sensors, and control the mechanical system to interact physically with the environment. Through human-machine-environment interaction, the efficiency of collaboration between humans and mechanical systems can be improved, making task completion faster and more efficient.

[0003] Human-machine-environment interaction mainly includes two parts: human-machine interaction and machine-object interaction. At present, human-machine interaction is mainly achieved through media such as vision and voice. These methods have certain requirements for the scene, which restricts its further development. With the increase in consumer demand for natural interaction, neural decoding technology has gradually become a widely-watched human-machine interaction technology. Neural decoding refers to the way of analyzing neural signals in human activities, including EEG signals and EMG signals, to obtain their movement intentions. Compared with other interaction methods, neural decoding is more concealed and natural, less constrained by the environment, and has great research and application value.

[0004] As a common medium for interaction between machines and the environment, robotic arms are widely used in industry, medicine, military and other fields. In these applications, robotic arms need to interact with the surrounding environment and other machines and equipment to complete various tasks accurately and efficiently. The current interaction mode mainly models the robotic arm through kinematics and dynamics, and controls it to move according to the motion trajectory set by the program. For example, Zheng Shengjie and others from the Shenzhen Institute of Advanced Technology of the Chinese Academy of Sciences used brain-computer interface technology to determine human intentions and send control instructions to a pre-set program to control the movement of the machine. However, this method requires the human nervous system to self-regulate information in real time to adjust the state of the machine, which increases the user's energy consumption and affects the interaction experience and efficiency. With the development of theories related to embodied intelligence, planning the motion strategy of the robotic arm in a data-driven way has gradually attracted attention. Especially with the development of large models in recent years, the information about the surrounding environment of the robotic arm can be input into the large model to plan the motion strategy, which can greatly reduce the energy consumed by users in human-computer interaction and has strong autonomy. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a robotic arm control method based on neural decoding and embodied intelligence, which can improve the accuracy of control instruction analysis results and the smoothness / flexibility of robotic arm movement, and enhance user experience.

[0006] The present invention provides a method for controlling a robotic arm based on neural decoding and embodied intelligence, comprising the following steps:

[0007] S1, collecting electromyographic signals of the user's limbs and filtering the electromyographic signals;

[0008] S2, calculating the time domain features between different channels of the electromyographic signal and reconstructing the frequency domain features between different channels;

[0009] S3, the time domain features and frequency domain features are fused and analyzed through a neural network to obtain the user's intention;

[0010] S4, collecting environmental information of the robot arm through sensors;

[0011] S5. Plan the motion trajectory of the robot arm according to the environmental information and user intention through the embodied large model.

[0012] Compared with the existing technology, the above technical solution can combine time domain features with frequency domain features to obtain user intentions, improve the accuracy of robotic arm control, and plan the motion trajectory of the robotic arm by embodying a large model, which can significantly improve the user experience.

[0013] In a possible implementation, step S1 includes the following steps:

[0014] S11, collecting electromyographic signals of the user's limbs through multi-channel electrodes;

[0015] S12, performing notch filtering, bandpass filtering and segmentation processing on the electromyographic signal in sequence.

[0016] Compared with the existing technology, the notch filtering using the above technical solution can effectively eliminate the noise caused by power frequency interference during the electromyography signal acquisition process; through bandpass filtering, it can reduce the noise caused by motion artifacts during the acquisition process and eliminate frequency domain components that are not very helpful for analyzing user intentions; through segmentation processing, the delay in actual scenarios can be reduced.

[0017] In a possible implementation manner, the segmentation process in step S12 specifically includes the following steps:

[0018] S121. Divide the electromyographic signal into segments every 150 ms.

[0019] Compared with the existing technology, the above technical solution can effectively reduce the delay in actual scenarios.

[0020] In a possible implementation, step S2 specifically includes the following steps:

[0021] S21, combining the electromyographic signals of different channels in pairs, and calculating the time domain characteristics of the electromyographic signals between every two channels after the combination by the following formula:

[0022]

[0023] Where N represents the length of the electromyographic signal, x i and i Respectively represent the i-th value of the signal segments of the two channels involved in the combination;

[0024] S22. The myoelectric signals between different channels are calculated by the following formula to obtain the spectrum diagrams between different channels:

[0025]

[0026] Among them, w represents the frequency value of the electromyographic signal, f(t) represents the time domain signal, and t represents time.

[0027] S23, arranging the frequency spectrum diagrams between different channels to obtain a frequency spectrum reconstruction diagram.

[0028] Compared with the existing technology, the adoption of the above technical solution can facilitate the combination of time domain features and frequency domain features in the subsequent process, improve the ability to characterize muscle coordination during human activities, and thus improve the accuracy.

[0029] In a possible implementation, step S3 specifically includes the following steps:

[0030] S31, constructing a feature fusion network, the feature fusion network including a frequency domain feature extraction module, a time domain feature extraction module, a first fully connected layer and a second fully connected layer;

[0031] S32, the spectrum reconstruction image is convolved and normalized by the frequency domain feature extraction module in sequence and then output, and the time domain feature is normalized by the time domain feature extraction module and then output;

[0032] S33, the output results of the frequency domain feature extraction module and the time domain feature extraction module are analyzed and processed in turn through the first fully connected layer and the second fully connected layer to obtain the user intention.

[0033] Compared with the existing technology, the above technical solution can fuse the time domain features and frequency domain features through a neural network, improve the characterization ability of muscle coordination during human activities, and thus obtain more accurate user intentions.

[0034] In a possible implementation, in step S4, environmental information is collected by using cameras with multiple angles.

[0035] Compared with the prior art, the above technical solution can effectively collect surrounding environmental information.

[0036] In a possible implementation, step S5 specifically includes the following steps:

[0037] S51, constructing a large embodied model;

[0038] S52. Plan the motion trajectory of the robot arm according to the environmental information and the user's intention through the embodied large model.

[0039] Compared with the existing technology, the above technical solution can use the embodied large model to plan the motion trajectory of the robotic arm, which is more efficient and can significantly improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the process of the present invention;

[0041] Figure 2 This is a schematic diagram of the neural network structure of the present invention;

[0042] Figure 3 This is a schematic diagram of calculating time domain features and frequency domain features in the present invention. DETAILED DESCRIPTION

[0043] First, those skilled in the art should understand that these implementations are only used to explain the technical principles of the embodiments of the present application, and are not intended to limit the protection scope of the embodiments of the present application. Those skilled in the art can make adjustments to them as needed to adapt to specific application scenarios.

[0044] The present application is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] The present application embodiment discloses a method for controlling a robotic arm based on neural decoding and embodied intelligence, comprising the following steps:

[0046] S1, collecting electromyographic signals of the user's limbs and filtering the electromyographic signals;

[0047] S2, calculating the time domain features between different channels of the electromyographic signal and reconstructing the frequency domain features between different channels;

[0048] S3, the time domain features and frequency domain features are fused and analyzed through a neural network to obtain the user's intention;

[0049] S4, collecting environmental information of the robot arm through sensors;

[0050] S5. Plan the motion trajectory of the robot arm according to the environmental information and user intention through the embodied large model.

[0051] Step S1 includes the following steps:

[0052] S11, collecting electromyographic signals of the user's limbs through multi-channel electrodes;

[0053] S12, performing notch filtering, bandpass filtering and segmentation processing on the electromyographic signal in sequence.

[0054] Specifically, collecting the electromyographic signals of the user's limbs requires multiple sets of electrodes placed in different parts of the body to collect signals, and each set of electrodes is a channel. Notch filtering is implemented using a notch filter, and bandpass filtering is implemented using a bandpass filter; segmentation processing can be implemented using a sliding window; notch filtering can effectively eliminate the noise caused by power frequency interference during the electromyographic signal collection process; bandpass filtering can reduce the noise caused by motion artifacts during the collection process, and eliminate frequency domain components that are not very helpful for analyzing user intentions. Segmentation processing can reduce the delay in actual scenarios.

[0055] The segmentation process in step S12 specifically includes the following steps:

[0056] S121. Divide the electromyographic signal into segments every 150 ms.

[0057] Step S2 specifically includes the following steps:

[0058] S21, combining the electromyographic signals of different channels in pairs, and calculating the time domain characteristics of the electromyographic signals between every two channels after the combination by the following formula:

[0059]

[0060] Where N represents the length of the electromyographic signal, x i and i Respectively represent the i-th value of the signal segments of the two channels involved in the combination;

[0061] S22. The myoelectric signals between different channels are calculated by the following formula to obtain the spectrum diagrams between different channels:

[0062]

[0063] The above formula is also the Fourier transform, where w represents the frequency value of the electromyographic signal, f(t) represents the time domain signal, and t represents time.

[0064] S23, arranging the frequency spectrum diagrams between different channels to obtain a frequency spectrum reconstruction diagram.

[0065] Specifically, in step S21, the electromyographic signals of different channels are combined in pairs, that is, channel 1 and channels 2, 3, 4, 5, 6, 7, 8 and 9 are combined in pairs without duplication, and the same is true for the remaining channels; Figure 3 As shown, in step S23, taking the total number of channels as 9 as an example, assuming that the original channel arrangement of the spectrum diagram is '123456789', taking channel 1 as an example, it can be seen that in this arrangement, channel 1 is only adjacent to channel 2, and the electrode channel arrangement after the arrangement is '1234567891357924681471582593694837261', the second channel 1 is adjacent to channel 3 and channel 9, the third channel 1 is adjacent to channel 4 and channel 8, and the fourth channel 1 is adjacent to channel 7 and channel 5 , the last channel 1 is adjacent to channel 6, that is, channel 1 is adjacent to the remaining channels 3, 4, 5, 6, 7, 8 and 9 once, and the remaining channels are similar, and no further details are given until the spectrum graph of each channel is adjacent to the spectrum graphs of other channels once, that is, the spectrum reconstruction graph is obtained; it should be noted that when the total number of channels is an even number, it may not be possible to ensure that the spectrum graph of each channel is adjacent to the spectrum graphs of other channels once, so it is necessary to try to ensure that the spectrum graph of each channel is adjacent to the spectrum graphs of other channels once.

[0066] The specific implementation code of the above arrangement algorithm is as follows:

[0067]

[0068] Step S3 specifically includes the following steps:

[0069] S31, constructing a feature fusion network, the feature fusion network including a frequency domain feature extraction module, a time domain feature extraction module, a first fully connected layer and a second fully connected layer;

[0070] S32, the spectrum reconstruction image is convolved and normalized by the frequency domain feature extraction module in sequence and then output, and the time domain feature is normalized by the time domain feature extraction module and then output;

[0071] S33, the output results of the frequency domain feature extraction module and the time domain feature extraction module are analyzed and processed in turn through the first fully connected layer and the second fully connected layer to obtain the user intention.

[0072] Specifically, as attached Figure 2As shown, the frequency domain feature extraction module includes two convolution layers and a normalization layer, and each convolution layer includes a 3×3 convolution layer, a normalization layer, and a 2×2 maximum pooling layer. After the spectrum reconstruction image is input into the frequency domain feature extraction module, it passes through the 3×3 convolution layer, the normalization layer, and the 2×2 maximum pooling layer of the first convolution layer in sequence, and then passes through the convolution, normalization, and maximum pooling processing in sequence, and then enters the second convolution layer and passes through the 3×3 convolution layer, the normalization layer, and the 2×2 maximum pooling layer of the first convolution layer in sequence, and then passes through the convolution, normalization, and maximum pooling processing in sequence, and finally expands and passes through the normalization layer for normalization and processing output. The time domain feature extraction module includes a normalization layer. After the time domain feature is input into the time domain feature extraction module, it passes through the normalization layer and is normalized and output. Finally, the output results of the frequency domain feature extraction module and the time domain feature extraction module pass through the first fully connected layer and the second fully connected layer in sequence for classification or regression algorithm analysis and processing to obtain the user intention.

[0073] In step S4, environmental information is collected by using cameras at multiple angles.

[0074] Step S5 specifically includes the following steps:

[0075] S51, constructing a large embodied model;

[0076] S52. Plan the motion trajectory of the robot arm according to the environmental information and the user's intention through the embodied large model.

[0077] Specifically, the equivalent concept corresponding to the embodied large model is the vision-action language model, which includes a visual encoder, a feature converter and a large language model. The large language model adopts Llama 2. In the paper "OpenVLA: An Open-Source Vision-Language-Action Model", there is a detailed record of adjusting Llama 2 through the OpenVLA method to plan the motion trajectory of the robot arm according to environmental information and user intentions, so as to illustrate that planning the motion trajectory of the robot arm according to environmental information and user intentions through the embodied large model is a prior art, so it will not be repeated in this application.

[0078] In the description of the present application, the description with reference to the terms "one embodiment", "some embodiments", "in the present embodiment", "specific example", or "some examples" etc. means that the specific features, mechanisms, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, mechanisms, materials or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0079] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A robotic arm control method based on neural decoding and embodied intelligence, characterized in that: The following steps are involved: S1, collecting electromyographic signals of the user's limbs, and filtering the electromyographic signals; S2, calculating the time domain features between different channels of the electromyographic signal and reconstructing the frequency domain features between different channels; S3, performing feature fusion processing and analysis on the time domain features and the frequency domain features through a neural network to obtain user intentions; S4, collecting environmental information of the robot arm through sensors; S5. Plan the motion trajectory of the robotic arm according to the environmental information and the user's intention through the embodied large model.

2. The method for controlling a robotic arm based on neural decoding and embodied intelligence according to claim 1, characterized in that: Step S1 includes the following steps: S11, collecting the electromyographic signals of the user's limbs through multi-channel electrodes; S12, performing notch filtering, bandpass filtering and segmentation processing on the electromyographic signal in sequence.

3. The method for controlling a robotic arm based on neural decoding and embodied intelligence according to claim 2, characterized in that: The segmentation process in step S12 specifically includes the following steps: S121, dividing the electromyographic signal into segments every 150 ms.

4. The method for controlling a robotic arm based on neural decoding and embodied intelligence according to claim 3, characterized in that: Step S2 specifically includes the following steps: S21, combining the electromyographic signals of different channels in pairs, and calculating the time domain characteristics of the electromyographic signals between every two channels after the combination by the following formula: Wherein, N represents the length of the electromyographic signal, x i and i Respectively represent the i-th value of the signal segments of the two channels involved in the combination; S22, the myoelectric signals between different channels are calculated by the following formula to obtain the frequency spectra between different channels: Wherein, w represents the frequency value of the electromyographic signal, f(t) represents the time domain signal, and t represents time. S23, arranging the frequency spectrum diagrams between different channels to obtain a frequency spectrum reconstruction diagram.

5. The method for controlling a robotic arm based on neural decoding and embodied intelligence according to claim 4, characterized in that: Step S3 specifically includes the following steps: S31, constructing a feature fusion network, wherein the feature fusion network includes a frequency domain feature extraction module, a time domain feature extraction module, a first fully connected layer, and a second fully connected layer; S32, the spectrum reconstruction image is subjected to convolution and normalization processing by the frequency domain feature extraction module in sequence and then output, and the time domain feature is subjected to normalization processing by the time domain feature extraction module and then output; S33, splicing the output results of the frequency domain feature extraction module and the time domain feature extraction module and sequentially passing them through the first fully connected layer and the second fully connected layer for analysis and processing to obtain the user intention.

6. The method for controlling a robotic arm based on neural decoding and embodied intelligence according to claim 5, characterized in that: In step S4, the environmental information is collected by using cameras at multiple angles.

7. The method for controlling a robotic arm based on neural decoding and embodied intelligence according to claim 6, characterized in that: Step S5 specifically includes the following steps: S51, constructing a large embodied model; S52: planning the motion trajectory of the robotic arm according to the environmental information and the user's intention through the embodied large model.

Images