Bionic manipulator control method, system and equipment based on sEMG signal and storage medium

Abstract

The invention discloses a bionic manipulator control method, system and device based on sEMG signals and a storage medium, and belongs to the technical field of bionic manipulators. The method comprises the steps that collected multi-modal signals are processed through a constructed multi-modal feature fusion neural network so as to recognize grabbing actions; the identification tag of the grabbing action is converted into a control signal of the bionic manipulator through a BP neural network; the multi-modal feature fusion neural network comprises an input layer, a feature extraction layer, a time sequence modeling layer, a feature fusion layer and an output layer; the feature extraction layer extracts local time sequence features of multi-modal signals, and the time sequence modeling layer models time sequence dependence of grabbing actions; a time-based attention mechanism is introduced into the feature fusion layer, the feature weight of each mode is adjusted in a self-adaptive mode, and then the features of the multiple modes are fused into a global feature vector. The recognition precision of the grabbing action can be improved, and therefore the control precision of the bionic manipulator is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of bionic manipulators, and in particular relates to a bionic manipulator control method, system, device and storage medium based on sEMG signals. Background Art

[0002] With the development of bionics and biomedical engineering technology, using bionic manipulators to help disabled people restore their daily functions has become a technical problem that needs to be solved urgently. In the design of bionic manipulators, it is necessary to consider the compactness and lightness of the structure, as well as to ensure that its operation is flexible and intelligent to adapt to various task requirements in daily life.

[0003] The core issue of achieving precise control of the robot is how to interpret the user's intention through a reliable signal source. Current research has tried a variety of control signal sources, including mechanical movement, voice control, tendon signals, brain waves, nerve signals, and surface electromyography (sEMG); among them, sEMG has gradually become the mainstream signal source in the field of robot prosthesis control due to its non-invasiveness, simple operation, and relatively low cost. sEMG signals can reflect the state of muscle activity in real time, capture the electrical signals stimulated by motor neurons, and provide effective support for the precise control of bionic robots; for some upper limb amputees, although they have lost their hand structure, their upper arm muscle tissue and nerve conduction function are still intact. The bioelectric signals generated by muscle contraction can reflect the amputee's movement intention to a high degree.

[0004] However, existing multi-freedom prosthetic hands still face challenges in terms of control accuracy, with a grasping success rate of only about 71%. The main reason is that the interpretation of multimodal bioelectric signals and the coordinated control of the bionic hand have not yet reached sufficient accuracy. Therefore, how to further improve the control accuracy and user experience of the bionic prosthetic hand is still the core research direction in the field of bionic prostheses. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a bionic manipulator control method, system, device and storage medium based on sEMG signals, which can improve the recognition accuracy of grasping actions, thereby improving the control accuracy of the bionic manipulator.

[0006] The present invention provides the following technical solutions:

[0007] In a first aspect, a bionic manipulator control method based on sEMG signals is provided, comprising:

[0008] Collecting multimodal signals; the multimodal signals include: sEMG signals and posture signals when the operator's arm grasps, and piezoelectric signals when the bionic manipulator grasps;

[0009] The multimodal signal is processed by constructing a multimodal feature fusion neural network to identify the grasping action; the multimodal feature fusion neural network includes an input layer, a feature extraction layer, a time series modeling layer, a feature fusion layer and an output layer; the input layer receives the multimodal signal, the feature extraction layer uses a convolutional neural network to extract the local time series features of the multimodal signal, and the time series modeling layer processes the extracted local time series features through a BiLSTM network to model the time series dependency of the grasping action; the feature fusion layer introduces a time-based attention mechanism to adaptively adjust the feature weights of each modality, and then fuses the multimodal features into a global feature vector; the output layer uses a classifier to identify the grasping action on the global feature vector and outputs the identification label of the grasping action;

[0010] The recognition labels of grasping actions are converted into control signals of bionic manipulators through BP neural network.

[0011] Optionally, the method further includes: performing multi-layer filtering on the collected sEMG signal of the operator's arm grasping, and the specific process is as follows:

[0012] The high-frequency and low-frequency noises of the sEMG signal were initially removed by bandpass filtering and envelope detection;

[0013] The sEMG signal after preliminary noise removal is decomposed into high-frequency IMF, low-frequency IMF and residual signal using EEMD decomposition method;

[0014] Determine whether the high-frequency IMF contains high-frequency noise. If not, directly use the noise-free high-frequency IMF to construct a new high-frequency IMF; if it exists, retain the noise-free IMF component, and denoise the noisy IMF component through the wavelet threshold, and combine the denoised IMF component with the retained noise-free IMF component to construct a new high-frequency IMF;

[0015] Reconstruct the new high-frequency IMF, the low-frequency IMF and the residual signal;

[0016] The reconstructed sEMG signal was subjected to short-time Fourier transform to remove periodic noise;

[0017] The sEMG signal after removing periodic noise was smoothed using a Savitzky-Golay filter to remove high-frequency fluctuation noise.

[0018] Optionally, the posture signal includes an angular velocity signal and an acceleration signal of the operator's arm;

[0019] When collecting the sEMG signal, preprocessing the sEMG signal, and judging whether the current sEMG signal is a grasping action according to the preprocessed sEMG signal, if not, directly outputting the preprocessed sEMG signal, and if so, obtaining the optimal grasping force according to the sEMG signal, and outputting the preprocessed sEMG signal;

[0020] When collecting the piezoelectric signal, the piezoelectric signal is preprocessed, and the current grasping force of the manipulator is obtained according to the preprocessed piezoelectric signal, and it is determined whether the current grasping force meets the value range of the optimal grasping force. If not, the grasping force transmitted by the piezoelectric signal is re-acquired until the value range of the optimal grasping force is met; if it is met, the optimal grasping force is output as the piezoelectric signal;

[0021] When collecting the posture signal, the posture signal is combined with the sEMG signal to determine whether it meets the preset arm movement posture requirements. If it meets the requirements, the currently collected posture signal is output; otherwise, the currently collected posture signal is removed.

[0022] Optionally, the feature fusion layer introduces a time-based attention mechanism to adaptively adjust the feature weights of each modality, thereby fusing the multi-modal features into a global feature vector. The process is as follows:

[0023] The eigenvectors of each modality are linearly transformed using the weight matrix that can be learned during the training process, and the eigenvectors of each modality after the linear transformation are concatenated to obtain the concatenated eigenvector H t ;

[0024] Based on the concatenated feature vector H t , through the nonlinear activation function, obtain the attention score e at each time step t ;

[0025] The attention score e of each time step is calculated through the Softmax function t Converted to the normalized attention weight α at each time step t ;

[0026] Normalize the attention weight α at each time step based on the obtained t , for all time steps, the concatenated feature vector H t Perform weighted summation to obtain the fused global feature vector.

[0027] Optionally, the linear transformation of the eigenvectors of each mode is expressed as:

[0028] h EMG =X EMG *W EMG

[0029] hMPU =X MPU *W MPU

[0030] h FSR =X FSR *W FSR

[0031] Among them, X EMG is the sEMG signal feature vector, W EMG For X EMG The corresponding learnable weight matrix; X MPU is the gesture signal feature vector, W MPU For X MPU The corresponding learnable weight matrix, X FSR is the piezoelectric signal eigenvector, W FSR For X FSR The corresponding learnable weight matrix; h EMG 、h MPU and h FSR are the sEMG signal feature vector, posture signal feature vector and piezoelectric signal feature vector mapped to the unified feature space;

[0032] The concatenated feature vector H t :

[0033] H t =concat(h EMG ,h MPU ,h FSR )

[0034] The attention score e at each time step t for:

[0035] e t =tanh(H t *W a )

[0036] Among them, W a are learnable parameters during the training process;

[0037] The normalized attention weight α at each time step t for:

[0038]

[0039] Where T is the total number of time steps;

[0040] The fused global feature vector X fuse for:

[0041]

[0042] Optionally, the converting the identification label of the grasping action into a control signal of the bionic manipulator through the BP neural network specifically includes:

[0043] The identification label of the grasping action is used as the input signal of the BP neural network, and the identification label of the grasping action is converted into the control signal of the bionic manipulator for output through the forward propagation of the BP neural network;

[0044] The forward propagation process of the BP neural network is:

[0045] h(t)=f(z(t))=f(U·h(t-1)+W·u(t)+b h )

[0046] y(t)=g(s(t))

[0047] s(t)=V·h(t)+b0

[0048] Among them, u(t) represents the input signal of the BP neural network, y(t) represents the output of the neurons in the output layer of the BP neural network, h(t) represents the output of the neurons in the hidden layer of the BP neural network, z(t) represents the input of the neurons in the hidden layer of the BP neural network, s(t) represents the input of the neurons in the output layer of the BP neural network, W represents the connection weight between the input layer and the hidden layer of the BP neural network; U represents the connection weight between the hidden layer and the receiving layer of the BP neural network; V represents the connection weight between the hidden layer and the output layer of the BP neural network, b h and b0 are the hidden layer and output layer biases respectively.

[0049] Optionally, the parameters of the BP neural network are adjusted online by an RTRL algorithm; the loss function L(t) of the RTRL algorithm is:

[0050]

[0051] e(t)=y d (t)-y(t)

[0052] Among them, y d (t) is the desired motion position of the manipulator, and y(t) is the actual output displacement of the manipulator;

[0053] The connection weights between the hidden layer and the receiving layer of the BP neural network change as follows:

[0054] U(t+1)=U(t)+ΔU(t)

[0055]

[0056] The connection weights between the input layer and the hidden layer of the BP neural network change as follows:

[0057] W(t+1)=W(t)+ΔW(t)

[0058]

[0059] The connection weights between the hidden layer and the output layer of the BP neural network change as follows:

[0060] V(t+1)=V(t)+ΔV(t)

[0061]

[0062] Among them, η1, η2 and η3 are the learning rates of parameter changes between the input layer and the hidden layer, the learning rates of parameter changes between the receiving layer and the hidden layer, and the learning rates of parameter changes between the output layer and the hidden layer, respectively.

[0063] In a second aspect, a bionic manipulator control system based on sEMG signals is provided, comprising:

[0064] A signal acquisition module, used for acquiring multimodal signals; the multimodal signals include: sEMG signals and posture signals when the operator's arm grasps, and piezoelectric signals when the bionic manipulator grasps;

[0065] An action recognition module is used to process multimodal signals through a constructed multimodal feature fusion neural network to identify grasping actions; the multimodal feature fusion neural network includes an input layer, a feature extraction layer, a time series modeling layer, a feature fusion layer and an output layer; the input layer receives multimodal signals, the feature extraction layer uses a convolutional neural network to extract local temporal features of the multimodal signals, and the time series modeling layer processes the extracted local temporal features through a BiLSTM network to model the temporal dependency of the grasping action; the feature fusion layer introduces a time-based attention mechanism to adaptively adjust the feature weights of each modality, and then fuse the multimodal features into a global feature vector; the output layer uses a classifier to identify the grasping action on the global feature vector and outputs an identification label of the grasping action;

[0066] The control signal generation module is used to convert the identification label of the grasping action into the control signal of the bionic manipulator through the BP neural network.

[0067] According to a third aspect, a computer device is provided, comprising a processor and a memory; wherein, when the processor executes a computer program stored in the memory, the steps of the bionic manipulator control method based on sEMG signals as described in any one of the first aspects are implemented.

[0068] In a fourth aspect, a computer-readable storage medium is provided for storing a computer program; when the computer program is executed by a processor, the steps of the bionic manipulator control method based on sEMG signals as described in any one of the first aspects are implemented.

[0069] Compared with the prior art, the present invention has the following beneficial effects:

[0070] (1) The multimodal feature fusion neural network constructed by the present invention performs feature vector recognition processing on the collected multimodal signals, thereby identifying common grasping actions; the multimodal feature fusion neural network of the present invention combines sEMG signals, posture signals and piezoelectric signals, uses convolutional neural networks for feature extraction, and uses long short-term memory networks for sequence modeling, ensuring better capture of complex time dependencies, focusing on important features by introducing an attention mechanism, and dynamically adjusting the degree of attention to different time steps or modal features, thereby improving recognition accuracy, especially in the case of data redundancy or large noise, improving the robustness and performance of the model, and achieving accurate classification of grasping actions.

[0071] (2) The present invention converts the identification label of the grasping action into the control signal of the bionic manipulator through the BP neural network, which reduces the computational complexity of the traditional neural network adaptive controller when adjusting the parameters, thereby further improving the control speed of the bionic manipulator. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is a control principle diagram of the bionic manipulator control method based on sEMG signals of the present invention;

[0073] Figure 2 It is a schematic diagram of the processing structure of the multi-modal signal acquisition of the present invention;

[0074] Figure 3 It is a structural block diagram of the sEMG signal acquisition module of the present invention;

[0075] Figure 4 It is a schematic diagram of the multi-layer filtering process of the sEMG signal of the present invention;

[0076] Figure 5 It is a schematic diagram of the position tracking of the bionic manipulator joint 1 according to the present invention;

[0077] Figure 6 It is a schematic diagram of the position tracking of the bionic manipulator joint 2 according to the present invention;

[0078] Figure 7 It is a schematic diagram of the angle tracking of the bionic manipulator joint 1 according to the present invention;

[0079] Figure 8 It is a schematic diagram of the angle tracking of the bionic manipulator joint 2 according to the present invention;

[0080] Fig. 9 It is a control input diagram of the joint 1 when the bionic manipulator moves according to the present invention;

[0081] Fig.10 It is the control input diagram of the joint 2 when the bionic manipulator moves according to the present invention;

[0082] Fig.11 It is a schematic diagram of the approximation result of sEMG signal by BP neural network adaptive controller signal. DETAILED DESCRIPTION

[0083] The present invention will be further described below in conjunction with the accompanying drawings. The following examples are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the scope of protection of the present invention. It should be noted that the term "comprising" and any variation of the description and claims of the present invention are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0084] Example 1

[0085] like Figure 1 As shown, a bionic manipulator control method based on sEMG signals is provided, comprising the following steps:

[0086] Step S1: Collect multimodal signals; the multimodal signals include: sEMG signals and posture signals when the operator's arm grasps, and piezoelectric signals when the bionic manipulator grasps.

[0087] like Figure 2 and Figure 3 As shown, step S1.1: acquisition of sEMG signals

[0088] The sEMG signal is collected by the sEMG signal acquisition module. Specifically, the sEMG signal acquisition module consists of an electrode sheet, a wire and a sEMG signal acquisition chip. The electrode sheet is attached to the surface of the arm muscle and is used to collect the electrical signal generated by muscle contraction. The electrode sheet is a fabric electrode sensor. One end of the wire is connected to three electrodes, of which the middle electrode sheet is used as the reference ground, and the left and right electrode sheets are used as differential signal input ends. The differential input signal is transmitted to the subsequent sEMG signal conditioning module through the wire for preprocessing the sEMG signal; the sEMG signal acquisition chip uses an ADC acquisition chip model ADS1292R. This chip is a 24-bit ADC chip specially used for collecting bioelectric signals. It has the advantages of high resolution, high precision and low noise, and can collect multi-channel synchronously, with powerful functions. At the same time, a Micro-SD card slot is configured to store the corresponding signal, which provides help for subsequent offline use and can be used for offline training of corresponding action recognition.

[0089] The sEMG signal conditioning module contains 10 independent conditioning circuits, each with a signal input, power interface, and signal output. This module is responsible for preamplifying the sEMG signal and improving the signal quality through denoising. Through this module, low-amplitude sEMG signals are enhanced and noise filtered, becoming more stable and reliable, and convenient for subsequent processing and analysis.

[0090] Determine whether the current sEMG signal is a grasping action based on the preprocessed sEMG signal. If not, directly output the preprocessed sEMG signal. If so, obtain the optimal grasping force based on the sEMG signal and output the preprocessed sEMG signal. Determine whether the sEMG signal is a grasping action by comparing the intensity or frequency of the sEMG signal with the set intensity or frequency.

[0091] As an option, during the sEMG signal acquisition process, the sampling rate is set to 1KHz, each sampling contains 2000 data points, and the terminal is configured as a differential input. Two sets of electrode patches for signal acquisition channels are attached to the three joints of the human arm, and there are a total of 6 sets of acquisition channels for the entire arm. During the experiment, the subject's left arm was kept naturally drooping and relaxed, and then five compound arm movements of pre-grasping, grasping, transporting and placing, letting go, and returning to the starting position were performed in sequence. In each action, 120 sets of signals were collected, and the first and last 10 sets were removed, and finally 100 sets of valid signal data were retained. In this way, 500 sets of sEMG signals will be obtained for each action for subsequent analysis and model training.

[0092] Step S1.2: Acquisition of piezoelectric signals

[0093] Normally, it is difficult to accurately judge the degree of contact between the bionic manipulator and the object based on sEMG signals alone. Therefore, the system introduces the pressure film sensor FSR402 to sense the contact force between the bionic manipulator and the object, and feeds it back to the control system to compare it with the optimal grasping force value of the bionic manipulator, thereby controlling the bionic manipulator to grasp objects of different sizes.

[0094] FSR402 is a piezoresistive pressure film sensor. The greater the contact force, the greater the detected voltage value and the smaller its internal resistance value. Therefore, by detecting its voltage value, the contact force value can be indirectly obtained.

[0095] When collecting piezoelectric signals, the piezoelectric signals are preprocessed, including amplification, compensation, etc.

[0096] The current grasping force of the manipulator is obtained according to the preprocessed piezoelectric signal, and it is determined whether the current grasping force meets the value range of the optimal grasping force. If not, the grasping force transmitted by the piezoelectric signal is re-acquired until it meets the value range of the optimal grasping force; if so, the optimal grasping force is output as a piezoelectric signal.

[0097] Step S1.3: Collection of attitude signals

[0098] The posture signal includes the angular velocity signal and acceleration signal of the operator's arm.

[0099] The posture signal is collected by the posture signal acquisition module. As the core part of motion detection, the posture signal acquisition module uses the module model MPU6050, which integrates a three-axis MEMS gyroscope and a three-axis MEMS accelerometer. It has the advantages of high precision, high response speed and low power consumption, and can monitor the angular velocity and acceleration changes of objects in real time. The posture signal acquisition module is pasted on the arm to enable it to capture the arm rotation inclination and motion information, and then connected to the main control chip through the IIC interface to package the inclination information and acceleration data for further motion analysis and control. After receiving the corresponding signal, the bionic manipulator drives each joint of the bionic manipulator through the motor to enable it to complete the corresponding grasping action.

[0100] When collecting posture signals, the posture signals are combined with sEMG signals to determine whether they meet the preset arm movement posture requirements. If they meet the requirements, the currently collected posture signals are output; otherwise, the currently collected posture signals are removed.

[0101] like Figure 4 As shown, step S1.4: filtering of sEMG signals

[0102] The sEMG signals collected when the operator's arm grasps are processed by multi-layer filtering. The specific process is as follows:

[0103] Step S1.4.1: Preliminarily remove high-frequency noise and low-frequency noise of the sEMG signal through bandpass filtering and envelope detection.

[0104] The design and selection of the bandpass filter directly affects the signal quality and the accuracy of subsequent analysis. Since the noise of the sEMG signal is mainly concentrated in the high-frequency band of the signal, the bandpass filter range is set to 20Hz to 450Hz to retain the effective frequency band of the electromyographic signal. This filtering method can filter out low-frequency artifacts concentrated in 0-20Hz (such as motion artifacts and baseline drift) and some high-frequency noise above 450Hz.

[0105] Envelope detection is used to extract the envelope of the signal, smooth the sEMG signal and eliminate the baseline drift, making the overall trend of the signal more obvious. The present invention adopts Hilbert transform to achieve this. Let the filtered signal be y(t), and its envelope e(t) can be obtained by Hilbert transform. The formula is:

[0106]

[0107] Step S1.4.2: Use the EEMD decomposition method to decompose the sEMG signal after preliminary noise removal into high-frequency IMF, low-frequency IMF and residual signal.

[0108] Step S1.4.3: Determine whether the high-frequency IMF contains high-frequency noise. If not, directly use the noise-free high-frequency IMF to construct a new high-frequency IMF; if it does, retain the noise-free IMF component, and denoise the noisy IMF component through the wavelet threshold, and combine the denoised IMF component and the retained noise-free IMF component to construct a new high-frequency IMF.

[0109] The EEMD decomposition method can refer to the existing technology; first use EEMD to decompose the signal into the inherent frequency components of the intrinsic mode function (EMD), and decompose the sEMG signal into multiple intrinsic mode functions (IMFs), each of which represents the components of the signal in different frequency ranges. High-frequency IMFs usually contain high-frequency noise, so these noise modes can be filtered out and useful low-frequency components can be retained. In actual operation, it is impossible to completely separate the real signal from the noise, so the wavelet transform is used to convert the signal from the time domain to the frequency domain, and the high-frequency IMF component is denoised. The wavelet threshold method is used to remove the noise component, and then the denoised EMD component and the high-frequency EMD component are superimposed and reconstructed to obtain the denoised sEMG signal. The frequency range of the high-frequency IMF and the low-frequency IMF can be determined based on experience.

[0110] Step S1.4.4: Reconstruct the new high-frequency IMF, low-frequency IMF and residual signal.

[0111] Step S1.4.5: Perform short-time Fourier transform on the reconstructed sEMG signal to remove periodic noise.

[0112] Since Fourier transform cannot well explain the disturbance of the mutation signal, short-time Fourier transform (STFT) is used to filter out periodic noise. This method converts the signal into the time-frequency domain, which is very suitable for analyzing non-stationary signals such as sEMG signals and can capture the time-frequency changes of sEMG signals in a finer granularity. At the same time, in the time-frequency diagram, frequency selection can be used to filter out high-frequency noise or periodic noise (such as power supply noise) within a specific time period.

[0113] Step S1.4.6: Use a Savitzky-Golay filter to smooth the sEMG signal after removing periodic noise to remove high-frequency fluctuation noise.

[0114] The Savitzky-Golay filter is a smoothing filtering technology that can smooth data and keep the shape of the original signal as much as possible through the principle of polynomial fitting. It is often used to smooth sharp noise and small random fluctuations in signals, while keeping the main trend and shape of the signal as much as possible. In the present invention, the filter is mainly used to remove random noise and high-frequency noise in the signal to make the signal smoother, while retaining the global trend and main features of the signal. It is used for envelope smoothing of sEMG signals to make the signal smoother and easier to analyze in subsequent processing stages.

[0115] The filtering of sEMG signals in the present invention combines the corresponding advantages of various filtering methods, removes different types of noise layer by layer, and innovatively combines various signal processing methods such as frequency domain, time domain and modal decomposition to form a comprehensive and hierarchical noise reduction framework, which can maximize the signal quality and reduce noise interference, maximize the retention of useful information of sEMG signals and improve the signal quality.

[0116] Step S2: Processing the multimodal signal through the constructed multimodal feature fusion neural network to identify the grasping action; the multimodal feature fusion neural network includes an input layer, a feature extraction layer, a time series modeling layer, a feature fusion layer and an output layer; the input layer receives the multimodal signal, the feature extraction layer uses a convolutional neural network to extract the local time series features of the multimodal signal, and the time series modeling layer processes the extracted local time series features through a BiLSTM network to model the time dependence of the grasping action; the feature fusion layer introduces a time-based attention mechanism to adaptively adjust the feature weights of each modality, and then fuse the multimodal features into a global feature vector; the output layer uses a classifier to identify the grasping action on the global feature vector, and outputs the identification label of the grasping action.

[0117] Specifically, the multi-modal feature fusion neural network (MFFNet) of the present invention includes an input layer, a feature extraction layer, a time series modeling layer, a feature fusion layer and an output layer.

[0118] Step S2.1: Input layer

[0119] The input layer is mainly used to extract the feature vectors of the collected signals, that is, for sEMG signals, the activity feature vectors of hand muscle movements are extracted; for MPU6050 gyroscope signals, the motion state signals of the hand, that is, the feature vectors of gyroscope inclination and acceleration are extracted; for FSR402 piezoelectric signals, the feature vectors of hand pressure signals in grasping movements are extracted. After obtaining these feature vectors, they are used as feature fusion layer vectors. Generating feature vectors for multiple signals can not only obtain the intention of muscle movements, but also understand the spatial motion state of the hand.

[0120] Step S2.2: Feature extraction layer

[0121] After obtaining the feature vectors of all collected signals, in order to further improve the accuracy of grasping action recognition, the present invention adopts a multimodal feature extraction strategy, uses a 1D convolutional neural network (CNN) to extract the local temporal features of the signal, performs dimensionality reduction compression through a pooling layer, retains the main features, and generates a new feature vector.

[0122] The process of multimodal data fusion is as follows: Feature extraction: All output signals of the input layer are used as input signals of the feature extraction layer, and a 1D convolutional neural network (CNN) is used to extract local temporal features from the denoised sEMG signal, posture signal and piezoelectric signal respectively; Dimensionality reduction and compression: After the output of the convolution layer, the dimension is reduced through the pooling layer, the main features are retained, and the data is output to the next layer.

[0123] Step S2.3: Time Series Modeling Layer

[0124] A bidirectional LSTM (BiLSTM) is established, and all the extracted multimodal feature data is input into the BiLSTM layer to model the temporal dependency of hand movements. BiLSTM can model temporal data from both the front and back directions, so it can capture the overall dynamic characteristics of the grasping action.

[0125] Step S2.4: Feature fusion layer

[0126] The sEMG signal, posture signal and piezoelectric signal are simply spliced ​​and weightedly fused to form a joint feature vector; this vector not only contains the muscle movement intention and movement state, but also adds real-time feedback of grasping pressure information.

[0127] At the same time, in the process of multimodal feature fusion, a time-based attention mechanism is added. The attention mechanism can enable the model to adaptively adjust the feature weights of different modalities according to the action relevance at each moment, giving priority to the most useful modal information at the current moment.

[0128] Specifically, it includes: Step S2.4.1: Use the weight matrix that can be learned during the training process to linearly transform the feature vectors of each mode, and concatenate the feature vectors of each mode after the linear transformation to obtain the concatenated feature vector H t ;

[0129] Using a learnable weight matrix W EMG , W MPU , W FSR Corresponding sEMG signal feature vector X EMG , attitude signal feature vector X MPU and the piezoelectric signal eigenvector X FSR , mapped to a unified feature space through these weight matrices. The specific calculation is as follows:

[0130] (1) Perform a linear transformation on the features of each mode to obtain a new representation:

[0131] h EMG =X EMG *W EMG

[0132] h MPU =X MPU *W MPU

[0133] h FSR =X FSR *W FSR

[0134] Among them, W EMG , W MPU , W FSR It is a learnable weight matrix that maps each modality feature to the same latent space dimension.

[0135] (2) These new representations h EMG 、h MPU 、h FSR Splice them together to get the spliced ​​feature vector H t :

[0136] H t =concat(h EMG ,h MPU ,h FSR ).

[0137] Step S2.4.2: Based on the concatenated feature vector H t , through the nonlinear activation function, obtain the attention score e at each time step t .

[0138] In order to calculate the weight of each modality, we need to get the attention score, which can be obtained through a weight matrix W a and a non-linear activation function (such as tanh) to achieve:

[0139] Specifically, the attention score is calculated:

[0140] e t =tanh(H t *W a )

[0141] Among them, W a is a learnable parameter, H t represents the fusion feature representation at time step t, e t is the attention score at each time step.

[0142] Step S2.4.3: Use the Softmax function to convert the attention score e of each time step into t Converted to normalized attention weight α at each time step t .

[0143]

[0144] Here, T is the total number of time steps.

[0145] Step S2.4.4: Normalize the attention weight α for each time step obtained t , for the concatenated feature vector H at all time steps t Perform weighted summation to obtain the fused global feature vector.

[0146] The fused global feature vector X fuse for:

[0147]

[0148] The final X fuse It is the result of weighted fusion of different modal features. The model will dynamically adjust according to the correlation of different modal features at different time steps to generate a global feature representation with adaptive capabilities.

[0149] The multimodal feature fusion neural network constructed by the present invention provides a more complete representation of grasping actions by means of multimodal data fusion; CNN is used to extract local features, and then LSTM is used to model time dependency, so that the whole system has powerful expression capabilities at both the feature level and the timing level; an attention mechanism is introduced to dynamically select the most critical time or modal features for action recognition, thereby improving the robustness and performance of the model, which is particularly suitable for the task of complex action sequence recognition; and weight adjustment is performed by using feature fusion that adapts to diversified actions, so that the model can adaptively adjust the weights of different modal features according to the needs of specific actions. This adaptive capability enables the model to automatically adjust the contribution of each modality according to different scenarios.

[0150] Step S2.5: Output layer

[0151] The fused feature vector is further reduced in feature dimension through a fully connected layer, and the Softmax classifier is used to classify and identify the action, and the identification label of the hand grasping action is output.

[0152] The multimodal fusion CNN of the present invention can adjust the grasping action strength in real time by fusing the spatial and temporal features of muscle signals and motion state information, and has closed-loop feedback capability, ensuring that the key information in each modal signal is effectively utilized, greatly improving the flexibility and stability of the grasping action. At the same time, the bidirectional LSTM is combined with the CNN, which not only retains the time dependence of the hand movement, but also captures the spatial local features in each modal signal. Finally, the importance of different modal features is dynamically adjusted through the attention mechanism, so that the model can automatically select the most useful features according to the specific action. The multimodal feature fusion neural network constructed by the present invention has a powerful classification ability: through multimodal data and temporal modeling, the model can accurately identify complex hand grasping actions (pre-grasping, grasping, transporting and placing, letting go and returning to the starting position).

[0153] Step S3: converting the recognition label of the grasping action into the control signal of the bionic manipulator through the BP neural network.

[0154] Traditional neural networks have a connection layer when they are designed. When designing a neural network adaptive controller, the added connection layer will bring more adaptive parameter adjustments, which increases the computational complexity of the neural network adaptive controller when adjusting parameters, thereby affecting the real-time control effect of the neural network adaptive controller. The present invention selects a BP neural network adaptive controller.

[0155] The identification label of the grasping action is used as the input signal of the BP neural network, and the identification label of the grasping action is converted into the control signal of the bionic manipulator for output through the forward propagation of the BP neural network.

[0156] The forward propagation process of the BP neural network is:

[0157] h(t)=f(z(t))=f(U·h(t-1)+W·u(t)+b h )

[0158] y(t)=g(s(t))

[0159] s(t)=V·h(t)+b0

[0160] Among them, u(t) represents the input signal of the BP neural network, y(t) represents the output of the neurons in the output layer of the BP neural network, h(t) represents the output of the neurons in the hidden layer of the BP neural network, z(t) represents the input of the neurons in the hidden layer of the BP neural network, s(t) represents the input of the neurons in the output layer of the BP neural network, W represents the connection weight between the input layer and the hidden layer of the BP neural network; U represents the connection weight between the hidden layer and the receiving layer of the BP neural network; V represents the connection weight between the hidden layer and the output layer of the BP neural network, b h and b0 are the hidden layer and output layer biases respectively.

[0161] The parameters of the BP neural network are adjusted online through the RTRL algorithm; the loss function L(t) of the RTRL algorithm is:

[0162]

[0163] e(t)=y d (t)-y(t)

[0164] Among them, y d (t) is the desired motion position of the manipulator, and y(t) is the actual output displacement of the manipulator;

[0165] The connection weights between the hidden layer and the receiving layer of the BP neural network change as follows:

[0166] U(t+1)=U(t)+ΔU(t)

[0167]

[0168] The connection weights between the input layer and the hidden layer of the BP neural network change as follows:

[0169] W(t+1)=W(t)+ΔW(t)

[0170]

[0171] The connection weights between the hidden layer and the output layer of the BP neural network change as follows:

[0172] V(t+1)=V(t)+ΔV(t)

[0173]

[0174] Among them, η1, η2 and η3 are the learning rates of parameter changes between the input layer and the hidden layer, the learning rates of parameter changes between the receiving layer and the hidden layer, and the learning rates of parameter changes between the output layer and the hidden layer, respectively.

[0175] The specific derivation process is as follows:

[0176]

[0177] Among them, f′(x)=f(x)·(1-f(x)); g′(x)=1.

[0178] remember We can get:

[0179]

[0180] remember We can get:

[0181]

[0182] like Figure 5-Figure 11 As shown, through the joint position tracking, joint angle tracking, joint control output and BP neural network adaptive controller approximation results of the sGME signal during the movement of the bionic manipulator, it can be seen that the recognition accuracy of the arm grasping action is greatly improved by the present invention, so that the bionic manipulator shows higher accuracy and stability in the recognition and execution of various complex actions.

[0183] The performance of the bionic manipulator of the present invention is evaluated by indicators such as accuracy, recall rate and F1-score.

[0184]

[0185] Among them, TP is a true positive example, TN is a true negative example, FP is a false positive example, and FN is a false negative example. The recognition accuracy of the system on five compound actions is 89.6%, showing good real-time and accuracy.

[0186] Example 2

[0187] Provided is a bionic manipulator control system based on sEMG signals, comprising a signal acquisition module, a motion recognition module and a control signal generation module; the motion recognition module and the control signal generation module of the invention are deployed on a host computer, the host computer is assembled with a bionic manipulator, the host computer recognizes hand movements in real time, and a bionic hand sample with five fingers and ten degrees of freedom is used to control the bionic manipulator to perform corresponding movements, thereby achieving high-precision motion imitation.

[0188] The signal acquisition module is used to collect multimodal signals; the multimodal signals include: sEMG signals and posture signals when the operator's arm grasps, and piezoelectric signals when the bionic manipulator grasps.

[0189] An action recognition module is used to process multimodal signals through a constructed multimodal feature fusion neural network to identify grasping actions; the multimodal feature fusion neural network includes an input layer, a feature extraction layer, a time series modeling layer, a feature fusion layer and an output layer; the input layer receives multimodal signals, the feature extraction layer uses a convolutional neural network to extract local temporal features of the multimodal signals, and the time series modeling layer processes the extracted local temporal features through a BiLSTM network to model the temporal dependency of the grasping action; the feature fusion layer introduces a time-based attention mechanism to adaptively adjust the feature weights of each modality, and then fuse the multimodal features into a global feature vector; the output layer uses a classifier to identify the grasping action on the global feature vector and outputs an identification label of the grasping action;

[0190] The control signal generation module is used to convert the identification label of the grasping action into the control signal of the bionic manipulator through the BP neural network.

[0191] For more specific processes of the above method, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0192] Example 3

[0193] A computer device comprises a processor and a memory; wherein, when the processor executes a computer program stored in the memory, the steps of the bionic manipulator control method based on sEMG signals described in any one of Embodiment 1 are implemented.

[0194] For more specific processes of the above method, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0195] Example 4

[0196] A computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, the steps of the bionic manipulator control method based on sEMG signals described in any one of Embodiment 1 are implemented.

[0197] For more specific processes of the above method, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0198] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system, device, and storage medium disclosed in the embodiment, since they correspond to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0199] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention or some parts of the embodiments.

[0200] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A bionic manipulator control method based on sEMG signals, characterized in that: include: Collecting multimodal signals; the multimodal signals include: sEMG signals and posture signals when the operator's arm grasps, and piezoelectric signals when the bionic manipulator grasps; The multimodal signal is processed by constructing a multimodal feature fusion neural network to identify the grasping action; the multimodal feature fusion neural network includes an input layer, a feature extraction layer, a time series modeling layer, a feature fusion layer and an output layer; the input layer receives the multimodal signal, the feature extraction layer uses a convolutional neural network to extract the local time series features of the multimodal signal, and the time series modeling layer processes the extracted local time series features through a BiLSTM network to model the time series dependency of the grasping action; the feature fusion layer introduces a time-based attention mechanism to adaptively adjust the feature weights of each modality, and then fuses the multimodal features into a global feature vector; the output layer uses a classifier to identify the grasping action on the global feature vector and outputs the identification label of the grasping action; The recognition labels of grasping actions are converted into control signals of bionic manipulators through BP neural network.

2. The bionic manipulator control method based on sEMG signals according to claim 1, characterized in that: The method also includes: performing multi-layer filtering on the sEMG signal collected when the operator's arm grasps, and the specific process is as follows: The high-frequency and low-frequency noises of the sEMG signal were initially removed by bandpass filtering and envelope detection; The sEMG signal after preliminary noise removal is decomposed into high-frequency IMF, low-frequency IMF and residual signal using EEMD decomposition method; Determine whether the high-frequency IMF contains high-frequency noise. If not, directly use the noise-free high-frequency IMF to construct a new high-frequency IMF; if it exists, retain the noise-free IMF component, and denoise the noisy IMF component through the wavelet threshold, and combine the denoised IMF component with the retained noise-free IMF component to construct a new high-frequency IMF; Reconstruct the new high-frequency IMF, the low-frequency IMF and the residual signal; The reconstructed sEMG signal was subjected to short-time Fourier transform to remove periodic noise; The sEMG signal after removing periodic noise was smoothed using a Savitzky-Golay filter to remove high-frequency fluctuation noise.

3. The bionic manipulator control method based on sEMG signals according to claim 1, characterized in that: The posture signal includes an angular velocity signal and an acceleration signal of the operator's arm; When collecting the sEMG signal, preprocessing the sEMG signal, and judging whether the current sEMG signal is a grasping action according to the preprocessed sEMG signal, if not, directly outputting the preprocessed sEMG signal, and if so, obtaining the optimal grasping force according to the sEMG signal, and outputting the preprocessed sEMG signal; When collecting the piezoelectric signal, the piezoelectric signal is preprocessed, and the current grasping force of the manipulator is obtained according to the preprocessed piezoelectric signal, and it is determined whether the current grasping force meets the value range of the optimal grasping force. If not, the grasping force transmitted by the piezoelectric signal is re-acquired until the value range of the optimal grasping force is met; if it is met, the optimal grasping force is output as the piezoelectric signal; When collecting the posture signal, the posture signal is combined with the sEMG signal to determine whether it meets the preset arm movement posture requirements. If it meets the requirements, the currently collected posture signal is output; otherwise, the currently collected posture signal is removed.

4. The bionic manipulator control method based on sEMG signals according to claim 1, characterized in that: The feature fusion layer introduces a time-based attention mechanism to adaptively adjust the feature weights of each modality, and then fuse the multi-modal features into a global feature vector. The process is as follows: The eigenvectors of each modality are linearly transformed using the weight matrix that can be learned during the training process, and the eigenvectors of each modality after the linear transformation are concatenated to obtain the concatenated eigenvector H t ; Based on the concatenated feature vector H t , through the nonlinear activation function, obtain the attention score e at each time step t ; The attention score e of each time step is calculated through the Softmax function t Converted to the normalized attention weight α at each time step t ; Normalize the attention weight α at each time step based on the obtained t , for all time steps, the concatenated feature vector H t Perform weighted summation to obtain the fused global feature vector.

5. The bionic manipulator control method based on sEMG signals according to claim 4, characterized in that: The linear transformation of the eigenvectors of each mode is expressed as: h EMG =X EMG *W EMG h MPU =X MPU *W MPU h FSR =X FSR *W FSR Among them, X EMG is the sEMG signal feature vector, W EMG For X EMG The corresponding learnable weight matrix; X MPU is the gesture signal feature vector, W MPU For X MPU The corresponding learnable weight matrix, X FSR is the piezoelectric signal eigenvector, W FSR For X FSR The corresponding learnable weight matrix; h EMG 、h MPU and h FSR are the sEMG signal feature vector, posture signal feature vector and piezoelectric signal feature vector mapped to the unified feature space; The concatenated feature vector H t : H t =concat(h EMG ,h MPU ,h FSR ) The attention score e at each time step t for: e t =tanh(H t *W a ) Among them, W a are learnable parameters during the training process; The normalized attention weight α at each time step t for: Where T is the total number of time steps; The fused global feature vector X fuse for:

6. The bionic manipulator control method based on sEMG signals according to claim 1, characterized in that: The method of converting the identification label of the grasping action into the control signal of the bionic manipulator through the BP neural network specifically includes: The identification label of the grasping action is used as the input signal of the BP neural network, and the identification label of the grasping action is converted into the control signal of the bionic manipulator for output through the forward propagation of the BP neural network; The forward propagation process of the BP neural network is: h(t)=f(z(t))=f(U·h(t-1)+W·u(t)+b h ) y(t)=g(s(t)) s(t)=V·h(t)+b0 Among them, u(t) represents the input signal of the BP neural network, y(t) represents the output of the neurons in the output layer of the BP neural network, h(t) represents the output of the neurons in the hidden layer of the BP neural network, z(t) represents the input of the neurons in the hidden layer of the BP neural network, s(t) represents the input of the neurons in the output layer of the BP neural network, W represents the connection weight between the input layer and the hidden layer of the BP neural network; U represents the connection weight between the hidden layer and the receiving layer of the BP neural network; V represents the connection weight between the hidden layer and the output layer of the BP neural network, b h and b0 are the hidden layer and output layer biases respectively.

7. The bionic manipulator control method based on sEMG signals according to claim 6, characterized in that: The parameters of the BP neural network are adjusted online through the RTRL algorithm; the loss function L(t) of the RTRL algorithm is: e(t)=y d (t)-y(t) Among them, y d (t) is the desired motion position of the manipulator, and y(t) is the actual output displacement of the manipulator; The connection weights between the hidden layer and the receiving layer of the BP neural network change as follows: U(t+1)=U(t)+ΔU(t) The connection weights between the input layer and the hidden layer of the BP neural network change as follows: W(t+1)=W(t)+ΔW(t) The connection weights between the hidden layer and the output layer of the BP neural network change as follows: V(t+1)=V(t)+ΔV(t) Among them, η1, η2 and η3 are the learning rates of parameter changes between the input layer and the hidden layer, the learning rates of parameter changes between the receiving layer and the hidden layer, and the learning rates of parameter changes between the output layer and the hidden layer, respectively.

8. A bionic manipulator control system based on sEMG signals, characterized in that: include: A signal acquisition module, used for acquiring multimodal signals; the multimodal signals include: sEMG signals and posture signals when the operator's arm grasps, and piezoelectric signals when the bionic manipulator grasps; An action recognition module is used to process multimodal signals through a constructed multimodal feature fusion neural network to identify grasping actions; the multimodal feature fusion neural network includes an input layer, a feature extraction layer, a time series modeling layer, a feature fusion layer and an output layer; the input layer receives multimodal signals, the feature extraction layer uses a convolutional neural network to extract local temporal features of the multimodal signals, and the time series modeling layer processes the extracted local temporal features through a BiLSTM network to model the temporal dependency of the grasping action; the feature fusion layer introduces a time-based attention mechanism to adaptively adjust the feature weights of each modality, and then fuse the multimodal features into a global feature vector; the output layer uses a classifier to identify the grasping action on the global feature vector and outputs an identification label of the grasping action; The control signal generation module is used to convert the identification label of the grasping action into the control signal of the bionic manipulator through the BP neural network.

9. A computer device, characterized in that: It comprises a processor and a memory; wherein, when the processor executes the computer program stored in the memory, the steps of the bionic manipulator control method based on sEMG signals described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: Used to store computer programs; when the computer programs are executed by the processor, the steps of the bionic manipulator control method based on sEMG signals described in any one of claims 1 to 7 are implemented.

Images