A sEMG signal-based bionic manipulator control method, system, device and storage medium

Abstract

The application discloses a kind of based on sEMG signal's bionic manipulator control method, system, equipment and storage medium, belong to the technical field of bionic manipulator, method includes by the multi-modal feature fusion neural network of construction to the multi-modal signal of acquisition is handled, to identify grasping action;Grasping action is identified label by BP neural network and is converted into the control signal of bionic manipulator;Multi-modal feature fusion neural network includes input layer, feature extraction layer, time series modeling layer, feature fusion layer and output layer;Feature extraction layer extracts the local time sequence feature of multi-modal signal, time series modeling layer models the time sequence dependency of grasping action;Feature fusion layer introduces time-based attention mechanism, adaptively adjusts the feature weight of each mode, and then the features of multi-modal are fused into global feature vector.The application can improve the recognition accuracy of grasping action, to improve the control accuracy of bionic manipulator.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of bionic manipulators, and particularly relates to a bionic manipulator control method, system, device and storage medium based on sEMG signals. BACKGROUND

[0002] With the development of bionics and biomedical engineering technology, using bionic manipulators to help disabled people restore daily functions has become a technical problem to be solved. In the design of bionic manipulators, not only the compactness and lightness of the structure should be considered, but also the flexibility and intelligence of the operation should be ensured to adapt to various task requirements in daily life.

[0003] The core problem of realizing precise control of the manipulator is how to interpret the user's intention through a reliable signal source. Current research has tried various control signal sources, including mechanical movement, sound control, tendon signal, brain wave, nerve signal and surface electromyography signal (sEMG). Among them, sEMG gradually becomes the mainstream signal source in the field of bionic hand control due to its non-invasiveness, simple operation and relatively low cost. sEMG signal can reflect the muscle activity state in real time and capture the electrical signal excited by motor neurons, providing effective support for the precise control of bionic manipulators. For some upper limb amputees, although the hand structure is lost, the muscle tissue and nerve conduction function of the upper arm are still intact. The bioelectric signal generated by muscle contraction can reflect the movement intention of the amputee.

[0004] However, the existing multi-degree-of-freedom artificial hand still faces challenges in control accuracy, with a success rate of only about 71%. The main reason is that the interpretation of multi-modal bioelectric signals and the coordinated control of bionic hands have not reached sufficient accuracy. Therefore, how to further improve the control accuracy and user experience of bionic hands is still the core research direction in the field of bionic prosthetics. SUMMARY

[0005] The application provides a bionic manipulator control method, system, device and storage medium based on sEMG signals, which can improve the recognition accuracy of grasping actions and thus improve the control accuracy of bionic manipulators.

[0006] The application provides the following technical solutions:

[0007] In a first aspect, a bionic manipulator control method based on sEMG signals is provided, comprising:

[0008] Collecting multi-modal signals; the multi-modal signals include sEMG signals and posture signals when the operator's arm is grasped, and piezoelectric signals when the bionic manipulator is grasped;

[0009] The multi-modal feature fusion neural network is used to process the multi-modal signals to identify the grabbing action; the multi-modal feature fusion neural network comprises an input layer, a feature extraction layer, a time sequence modeling layer, a feature fusion layer and an output layer; the input layer receives the multi-modal signals, the feature extraction layer extracts local time sequence features of the multi-modal signals by using a convolutional neural network, the time sequence modeling layer processes the extracted local time sequence features by using a BiLSTM network to model the time sequence dependency of the grabbing action; the feature fusion layer introduces a time-based attention mechanism to adaptively adjust the feature weights of each mode, and then fuses the features of the multi-modal signals into a global feature vector; and the output layer uses a classifier to identify the global feature vector to identify the grabbing action and outputs an identification label of the grabbing action.

[0010] The identification label of the grabbing action is converted into a control signal of the bionic manipulator by using a BP neural network.

[0011] Optionally, the sEMG signals of the operator's arm during grabbing are further subjected to multi-layer filtering, and the specific process is as follows:

[0012] The sEMG signals are subjected to band-pass filtering and envelope detection to preliminarily remove high-frequency noise and low-frequency noise of the sEMG signals;

[0013] The sEMG signals after preliminary noise removal are decomposed into high-frequency IMFs, low-frequency IMFs and residual signals by using an EEMD decomposition method;

[0014] It is judged whether the high-frequency IMFs contain high-frequency noise, if not, the high-frequency IMFs without noise are directly used to construct new high-frequency IMFs; if yes, the IMF components without noise are retained, and the IMF components with noise are denoised by using a wavelet threshold, and the denoised IMF components and the retained IMF components without noise are combined to construct new high-frequency IMFs;

[0015] The new high-frequency IMFs, the low-frequency IMFs and the residual signals are subjected to signal reconstruction;

[0016] The reconstructed sEMG signals are subjected to short-time Fourier transform to remove periodic noise;

[0017] The sEMG signals after removal of periodic noise are subjected to smoothing processing by using a Savitzky-Golay filter to remove high-frequency fluctuation noise.

[0018] Optionally, the posture signals comprise angular velocity signals and acceleration signals of the operator's arm.

[0019] When the sEMG signal is collected, the sEMG signal is preprocessed, and whether the current sEMG signal is a grasping action is determined according to the preprocessed sEMG signal, if not, the preprocessed sEMG signal is directly output, if yes, the optimal grasping force is obtained according to the sEMG signal, and the preprocessed sEMG signal is output;

[0020] When the piezoelectric signal is collected, the piezoelectric signal is preprocessed, the current grasping force of the manipulator is obtained according to the preprocessed piezoelectric signal, and whether the current grasping force meets the value range of the optimal grasping force is determined, if not, the grasping force transmitted by the piezoelectric signal is reacquired until the value range of the optimal grasping force is met; if yes, the optimal grasping force is output as the piezoelectric signal;

[0021] When the posture signal is collected, whether the posture signal meets the preset arm action posture requirement is determined in combination with the sEMG signal, if yes, the current collected posture signal is output, otherwise, the current collected posture signal is removed.

[0022] Optionally, the feature fusion layer introduces a time-based attention mechanism to adaptively adjust the feature weights of each modality, and then fuses the features of multiple modalities into a global feature vector, the process is as follows:

[0023] The feature vectors of each modality are linearly transformed using a weight matrix that can be learned during training, and the feature vectors of each modality after linear transformation are spliced to obtain a spliced feature vector H t ;

[0024] Based on the spliced feature vector H t , the attention score e t of each time step is obtained through a nonlinear activation function;

[0025] The attention score e t of each time step is converted into the normalized attention weight a t of each time step through a Softmax function;

[0026] Based on the obtained normalized attention weight a t of each time step, the spliced feature vector H t of all time steps is weighted and summed to obtain the fused global feature vector.

[0027] Optionally, the linear transformation of the feature vectors of each modality is represented as:

[0028] h EMG =X EMG *W EMG

[0029] hMPU = X MPU * W MPU

[0030] h FSR = X FSR * W FSR

[0031] wherein X EMG is the sEMG signal feature vector, W EMG is the corresponding learnable weight matrix of X EMG , X MPU is the posture signal feature vector, W MPU is the corresponding learnable weight matrix of X MPU , X FSR is the piezoelectric signal feature vector, W FSR is the corresponding learnable weight matrix of X FSR ; h EMG , h MPU and h FSR are the sEMG signal feature vector, the posture signal feature vector and the piezoelectric signal feature vector mapped to the unified feature space;

[0032] The concatenated feature vector H t is:

[0033] H t = concat(h EMG , h MPU , h FSR )

[0034] The attention score e t of each time step is:

[0035] e t = tanh(H t * W a )

[0036] wherein W a is the learnable parameter in the training process;

[0037] The normalized attention weight a t of each time step is:

[0038]

[0039] wherein T is the total number of time steps;

[0040] The fused global feature vector X fuse is:

[0041]

[0042] Optionally, the BP neural network converts the recognition label of the grabbing action into the control signal of the bionic manipulator, and specifically comprises:

[0043] The recognition label of the grabbing action is taken as an input signal of the BP neural network, and the recognition label of the grabbing action is converted into the control signal of the bionic manipulator through forward propagation of the BP neural network and then outputted;

[0044] The forward propagation process of the BP neural network is as follows:

[0045] h(t)=f(z(t))=f(U·h(t-1)+W·u(t)+b h )

[0046] y(t)=g(s(t))

[0047] s(t)=V·h(t)+b0

[0048] Wherein, u(t) represents the input signal of the BP neural network, y(t) represents the output of the output layer neuron of the BP neural network, h(t) represents the output of the hidden layer neuron of the BP neural network, z(t) represents the input of the hidden layer neuron of the BP neural network, s(t) represents the input of the output layer neuron of the BP neural network, W represents the connection weight value between the input layer and the hidden layer of the BP neural network; U represents the connection weight value between the hidden layer and the output layer of the BP neural network; V represents the connection weight value between the hidden layer and the output layer of the BP neural network, b h and b0 are the bias of the hidden layer and the output layer respectively.

[0049] Optionally, the parameters of the BP neural network are adjusted online through the RTRL algorithm; the loss function L(t) of the RTRL algorithm is as follows:

[0050]

[0051] e(t)=y d (t)-y(t)

[0052] Wherein, y d (t) is the expected motion position of the manipulator, and y(t) is the actual output displacement of the manipulator.

[0053] The connection weight value between the hidden layer and the output layer of the BP neural network changes as follows:

[0054] U(t+1)=U(t)+ΔU(t)

[0055]

[0056] The connection weight value between the input layer and the hidden layer of the BP neural network changes as follows:

[0057] W(t+1) = W(t) + AW(t)

[0058]

[0059] The connection weight value between the hidden layer and the output layer of the BP neural network changes as follows:

[0060] V(t+1) = V(t) + AV(t)

[0061]

[0062] wherein, η1, η2 and η3 are learning rates of parameter changes between the input layer and the hidden layer, between the hidden layer and the output layer and between the hidden layer and the output layer respectively.

[0063] In a second aspect, a bionic manipulator control system based on sEMG signals is provided, comprising:

[0064] A signal acquisition module is configured to acquire multi-modal signals, wherein the multi-modal signals include sEMG signals and posture signals when an operator's arm is grabbing, and piezoelectric signals when a bionic manipulator is grabbing.

[0065] An action recognition module is configured to process the multi-modal signals through a constructed multi-modal feature fusion neural network to recognize a grabbing action, wherein the multi-modal feature fusion neural network includes an input layer, a feature extraction layer, a time sequence modeling layer, a feature fusion layer and an output layer; the input layer receives the multi-modal signals, the feature extraction layer extracts local time sequence features of the multi-modal signals by using a convolutional neural network, the time sequence modeling layer processes the extracted local time sequence features through a BiLSTM network to model time sequence dependencies of the grabbing action, the feature fusion layer introduces a time-based attention mechanism to adaptively adjust feature weights of each modality, and then fuses the features of the multi-modal into a global feature vector; and the output layer uses a classifier to recognize the global feature vector to recognize the grabbing action and outputs a recognition label of the grabbing action.

[0066] A control signal generation module is configured to convert the recognition label of the grabbing action into a control signal of the bionic manipulator through a BP neural network.

[0067] In a third aspect, a computer device is provided, comprising a processor and a memory; wherein the processor implements the steps of the bionic manipulator control method based on sEMG signals according to any one of the first aspect when executing a computer program saved in the memory.

[0068] In a fourth aspect, a computer readable storage medium for storing a computer program is provided, and the computer program is executed by a processor to implement the steps of the sEMG signal based bionic robot control method of any one of the first aspect.

[0069] Compared with the prior art, the present application has the following beneficial effects:

[0070] (1) The multi-modal feature fusion neural network constructed by the present application performs feature vector recognition processing on the collected multi-modal signals, thereby recognizing common grasping actions; the multi-modal feature fusion neural network of the present application combines sEMG signals, posture signals and piezoelectric signals, uses a convolutional neural network for feature extraction, uses a long short-term memory network for sequence modeling, ensures better capture of complex time-dependent relationships, focuses on important features by introducing an attention mechanism, dynamically adjusts the attention degree to different time steps or modal features, thereby improving the recognition accuracy, especially in the case of data redundancy or high noise, improving the robustness and performance of the model, and realizing accurate classification of grasping actions.

[0071] (2) The present application converts the recognition label of the grasping action into a control signal of the bionic robot through the BP neural network, reduces the calculation complexity of the traditional neural network adaptive controller when adjusting parameters, and thereby can further improve the control speed of the bionic robot. BRIEF DESCRIPTION OF DRAWINGS

[0072] Figure 1 is the control principle diagram of the sEMG signal based bionic robot control method of the present application;

[0073] Figure 2 is the processing structure schematic diagram when the multi-modal signal of the present application is collected;

[0074] Figure 3 is the structure block diagram of the sEMG signal acquisition module of the present application;

[0075] Figure 4 is the multi-layer filtering flowchart of the sEMG signal of the present application;

[0076] Figure 5 is the position tracking schematic diagram of the bionic robot joint 1 of the present application;

[0077] Figure 6 is the position tracking schematic diagram of the bionic robot joint 2 of the present application;

[0078] Figure 7 is the angle tracking schematic diagram of the bionic robot joint 1 of the present application;

[0079] Figure 8 is the angle tracking schematic diagram of the bionic robot joint 2 of the present application;

[0080] Figure 9 is the control input graph of joint 1 when the bionic manipulator is moving;

[0081] Figure 10 is the control input graph of joint 2 when the bionic manipulator is moving;

[0082] Figure 11 is the schematic diagram of the approximation result of the sEMG signal by the BP neural network adaptive controller signal. DETAILED DESCRIPTION

[0083] The application will be further described below in conjunction with the drawings. The following examples are only used to more clearly illustrate the technical solutions of the application, and cannot be used to limit the protection scope of the application. It should be noted that the terms "comprise" and any variations thereof in the specification and claims of the application are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0084] Example 1

[0085] As shown in Figure 1 , a bionic manipulator control method based on sEMG signals is provided, comprising the following steps:

[0086] Step S1: Collecting multi-modal signals; the multi-modal signals include: sEMG signals and posture signals when the operator's arm is grabbing, and piezoelectric signals when the bionic manipulator is grabbing.

[0087] As shown in Figure 2 and Figure 3 , step S1.1: sEMG signal acquisition

[0088] The sEMG signal is collected by an sEMG signal collection module. Specifically, the sEMG signal collection module includes electrode pieces, wires, and an sEMG signal collection chip. The electrode pieces are attached to the surface of the arm muscles and are used to collect the electrical signals generated by muscle contraction. The electrode pieces are fabric electrode sensors. One end of the wire is connected to three electrode pieces, with the middle electrode piece acting as a reference ground and the left and right electrode pieces acting as differential signal input ends. The differential input signal is transmitted to the subsequent sEMG signal conditioning module through the wire for pre-processing of the sEMG signal. The sEMG signal collection chip is an ADC collection chip with a model number of ADS1292R. This chip is a 24-bit ADC chip specifically designed for collecting bioelectric signals, with the advantages of high resolution, high precision, and low noise. It can also perform multi-channel synchronous collection and has powerful functions. A Micro-SD card slot is also configured to store the corresponding signals, which can help with offline use and can be used for offline training of corresponding motion recognition.

[0089] The sEMG signal conditioning module includes 10 independent conditioning circuits, each with a signal input end, a power interface, and a signal output end. This module is responsible for pre-amplifying the sEMG signal and improving signal quality through noise reduction processing. Through this module, low-amplitude sEMG signals are enhanced and noise-filtered, making them more stable and reliable for subsequent processing and analysis.

[0090] According to the pre-processed sEMG signal, it is determined whether the current sEMG signal is a grasping action. If not, the pre-processed sEMG signal is directly output. If so, the optimal grasping force is obtained based on the sEMG signal, and the pre-processed sEMG signal is output. The strength or frequency of the sEMG signal is compared with the set strength or frequency to determine whether the sEMG signal is a grasping action.

[0091] As an option, during sEMG signal collection, the sampling rate is set to 1 KHz, each sample contains 2000 data points, and the connection end is configured as a differential input. At the three joint parts of the human arm, two groups of signal collection channel electrode pieces are attached, and the entire arm has six groups of collection channels. During the experiment, the subject's left arm is kept naturally drooping and relaxed, and then performs five compound arm actions in sequence: pre-grasping, grasping, carrying and placing, releasing the hand, and returning to the starting position. Under each action, 120 groups of signals are collected, and the first and last 10 groups are removed, finally retaining 100 groups of valid signal data. In this way, each action will obtain 500 groups of sEMG signals for subsequent analysis and model training.

[0092] Step S1.2: Collection of Piezoelectric Signal

[0093] Generally, it is difficult to accurately determine the contact degree between the bionic robot hand and the object only by the sEMG signal, therefore, the system introduces a pressure film sensor FSR402 to perceive the contact force value between the bionic robot hand and the object, and feeds back to the control system to compare with the optimal grasping force value of the bionic robot hand, and then controls the bionic robot hand to grasp objects of different sizes.

[0094] FSR402 is a piezoresistive pressure film sensor, the greater the contact force, the greater the detected voltage value, and the smaller the internal resistance value, so by detecting the voltage value, the contact force value is indirectly obtained.

[0095] When collecting the piezoelectric signal, the piezoelectric signal is preprocessed, including amplification, compensation, etc.

[0096] According to the preprocessed piezoelectric signal, the current grasping force of the robot hand is obtained, and it is judged whether the current grasping force meets the value range of the optimal grasping force, if not, the grasping force transmitted by the piezoelectric signal is reacquired until the value range of the optimal grasping force is met; if it is met, the optimal grasping force is taken as the piezoelectric signal output.

[0097] Step S1.3: Acquisition of attitude signal

[0098] The attitude signal includes the angular velocity signal and the acceleration signal of the operator's arm.

[0099] The attitude signal is collected by an attitude signal acquisition module, which is the core part of motion detection. The module model is MPU6050, which integrates a three-axis MEMS gyroscope and a three-axis MEMS accelerometer, has the advantages of high precision, high response speed and low power consumption, and can monitor the angular velocity and acceleration changes of the object in real time. The attitude signal acquisition module is pasted on the arm, so that it can capture the rotation angle and motion information of the arm, and then connect with the master control chip through the IIC interface, package and send the angle information and acceleration data for further motion analysis and control. After receiving the corresponding signal, the bionic robot hand drives each joint of the bionic robot hand through the motor, so that it completes the corresponding grasping action.

[0100] When collecting the attitude signal, the attitude signal is combined with the sEMG signal to determine whether it meets the preset arm action attitude requirement, if it meets, the currently collected attitude signal is output, otherwise, the currently collected attitude signal is removed.

[0101] As shown in Figure 4 Step S1.4: filtering processing of sEMG signal

[0102] The collected sEMG signal of the operator's arm during grasping is subjected to multi-layer filtering processing, and the specific process is as follows:

[0103] Step S1.4.1: Preliminary removal of high-frequency noise and low-frequency noise of sEMG signal by band-pass filtering and envelope detection.

[0104] The design and selection of the band-pass filter directly affect the signal quality and the accuracy of subsequent analysis. Since the noise of sEMG signal is mainly concentrated in the high-frequency band of the signal, the band-pass filtering range is set to 20-450 Hz to retain the effective frequency band of the electromyographic signal. This filtering method can filter out low-frequency artifacts (such as motion artifacts and baseline drift) concentrated in 0-20 Hz and part of high-frequency noise above 450 Hz.

[0105] Envelope detection is used to extract the envelope of the signal, smooth the sEMG signal and eliminate baseline drift, making the overall trend of the signal more obvious. The present application uses Hilbert transform to achieve this, assuming that the filtered signal is y(t), its envelope e(t) can be obtained by Hilbert transform The formula is:

[0106]

[0107] Step S1.4.2: Use EEMD decomposition method to decompose the sEMG signal after preliminary noise removal into high-frequency IMF, low-frequency IMF and residual signal.

[0108] Step S1.4.3: Determine whether the high-frequency IMF contains high-frequency noise. If not, directly use the high-frequency IMF without noise to construct a new high-frequency IMF; if so, retain the IMF component without noise, and denoise the IMF component containing noise by wavelet threshold, and combine the denoised IMF component and the retained IMF component without noise to construct a new high-frequency IMF.

[0109] The EEMD decomposition method can refer to the prior art; first, use EEMD to decompose the signal into intrinsic frequency components of the intrinsic mode function (EMD), and decompose the sEMG signal into multiple intrinsic mode functions (IMF), each IMF represents the component of the signal in different frequency range. High-frequency IMF usually contains high-frequency noise, so these noise modes can be filtered out to retain useful low-frequency components. In actual operation, it is impossible to completely separate the real signal and noise, so the signal is converted from time domain to frequency domain by wavelet transform, the high-frequency IMF component is denoised, the noise component is removed by wavelet threshold method, and then the denoised EMD component and high-frequency EMD component are superimposed to reconstruct the denoised sEMG signal. The frequency range of high-frequency IMF and low-frequency IMF can be determined empirically.

[0110] Step S1.4.4: Signal reconstruction of new high-frequency IMF, low-frequency IMF and residual signal.

[0111] Step S1.4.5: Perform short-time Fourier transform on the reconstructed sEMG signal to remove periodic noise.

[0112] Since Fourier transform cannot well explain the disturbance of the mutation signal, short-time Fourier transform STFT is adopted to filter out periodic noise. This method converts the signal into a time-frequency domain and is very suitable for analyzing non-stationary signals such as sEMG signals. It can capture the time-frequency changes of the sEMG signal in a more fine-grained manner. At the same time, in the time-frequency diagram, frequency selection can be used to filter out high-frequency noise or periodic noise (such as power noise) in a specific time period.

[0113] Step S1.4.6: Smooth the sEMG signal after removing periodic noise using a Savitzky-Golay filter to remove high-frequency fluctuation noise.

[0114] The Savitzky-Golay filter is a smoothing filter technology that can smooth data while preserving the original signal shape through polynomial fitting. It is commonly used to smooth sharp noise and small random fluctuations in signals while preserving the main trend and shape of the signal. In this invention, the filter is mainly used to remove random noise and high-frequency noise in the signal, making the signal smoother while preserving the overall trend and main features of the signal. It is used for envelope smoothing of sEMG signals, making the signal smoother and easier to analyze in the subsequent processing stage.

[0115] The filtering of sEMG signals in this invention combines the advantages of various filtering methods, removing different types of noise layer by layer. It innovatively combines frequency domain, time domain, and modal decomposition, forming a comprehensive and hierarchical noise reduction framework that can maximize signal quality and reduce noise interference, while maximizing the preservation of useful information in sEMG signals and improving signal quality.

[0116] Step S2: Process the multi-modal signal through the constructed multi-modal feature fusion neural network to identify the grasping action; the multi-modal feature fusion neural network includes an input layer, a feature extraction layer, a time series modeling layer, a feature fusion layer, and an output layer; the input layer receives the multi-modal signal, the feature extraction layer extracts the local time sequence features of the multi-modal signal using a convolutional neural network, the time series modeling layer processes the extracted local time sequence features through a BiLSTM network to model the time sequence dependency of the grasping action; the feature fusion layer introduces a time-based attention mechanism to adaptively adjust the feature weights of each modality, and then fuses the multi-modal features into a global feature vector; the output layer uses a classifier to identify the global feature vector for the grasping action and outputs the identification label of the grasping action.

[0117] Specifically, the multi-modal feature fusion neural network (MFFNet) of the present application comprises an input layer, a feature extraction layer, a time series modeling layer, a feature fusion layer and an output layer.

[0118] Step S2.1: input layer

[0119] The input layer is mainly used to extract the feature vectors of the collected signals, i.e., for sEMG signals, the activity feature vectors of hand muscle movements are extracted; for MPU6050 gyroscope signals, the feature vectors of the motion state signals of the hand, i.e., the inclination and acceleration of the gyroscope, are extracted; for FSR402 piezoelectric signals, the feature vectors of the hand pressure signals in the grasping action are extracted. After obtaining these feature vectors, they are taken as feature fusion layer vectors, and generating feature vectors for multiple signals can not only obtain muscle movement intentions, but also understand the spatial motion state of the hand.

[0120] Step S2.2: feature extraction layer

[0121] After obtaining the feature vectors of all collected signals, in order to further improve the accuracy of grasping action recognition, the present application adopts a multi-modal feature extraction strategy, uses a 1D convolutional neural network (CNN) to extract local time sequence features of the signals, and performs dimensionality reduction compression through a pooling layer to retain the main features and generate new feature vectors.

[0122] The process of multi-modal data fusion is as follows: feature extraction: all output signals of the input layer are taken as input signals of the feature extraction layer, a 1D convolutional neural network (CNN) is used to extract local time sequence features from the denoised sEMG signals, posture signals and piezoelectric signals respectively; dimensionality reduction compression: after the convolutional layer outputs, dimensionality reduction is performed through the pooling layer to retain the main features and output the data to the next layer.

[0123] Step S2.3: time series modeling layer

[0124] A bidirectional LSTM (BiLSTM) is established, and all multi-modal feature data extracted is input into the BiLSTM layer for modeling the time sequence dependency of hand movements. BiLSTM can model time sequence data from both forward and backward directions, so it can capture the overall dynamic features of the grasping action.

[0125] Step S2.4: feature fusion layer

[0126] The sEMG signals, posture signals and piezoelectric signals are simply spliced and weightedly fused to form a joint feature vector; this vector not only contains muscle movement intentions and motion states, but also adds real-time feedback of grasping pressure information.

[0127] Meanwhile, in the process of multi-modal feature fusion, a time-based attention mechanism is added. The attention mechanism can adjust the feature weights of different modalities adaptively according to the action relevance of each time, and give priority to the most useful modal information at the current time.

[0128] Specifically, it includes: step S2.4.1: using the learnable weight matrix in the training process to linearly transform the feature vectors of each modality, and splicing the linearly transformed feature vectors of each modality to obtain the spliced feature vector H t ;

[0129] The learnable weight matrix W EMG , W MPU , W FSR is used to map the sEMG signal feature vector X EMG , the posture signal feature vector X MPU and the piezoelectric signal feature vector X FSR to a unified feature space. The specific calculation is as follows:

[0130] (1) Linearly transform the features of each modality to obtain new representations:

[0131] h EMG =X EMG *W EMG

[0132] h MPU =X MPU *W MPU

[0133] h FSR =X FSR *W FSR

[0134] Where W EMG , W MPU , W FSR are learnable weight matrices that map the features of each modality to the same hidden space dimension.

[0135] (2) Splice these new representations h EMG , h MPU , h FSR to obtain the spliced feature vector H t :

[0136] H t =concat(h EMG ,h MPU ,h FSR ).

[0137] Step S2.4.2: Based on the spliced feature vector H t , obtain the attention score e t of each time step through a nonlinear activation function.

[0138] In order to calculate the weight of each modality, the attention score needs to be obtained, which can be achieved through a weight matrix W a and a nonlinear activation function (such as tanh):

[0139] Specifically, the attention score is calculated as:

[0140] e t = tanh(H t * W a )

[0141] where W a is a learnable parameter, H t represents the fused feature representation at time step t, and e t is the attention score of each time step.

[0142] Step S2.4.3: Convert the attention score e t of each time step to the normalized attention weight a t of each time step through the Softmax function.

[0143]

[0144] where T is the total number of time steps.

[0145] Step S2.4.4: Based on the obtained normalized attention weight a t of each time step, perform weighted summation on the spliced feature vector H t at all time steps to obtain the fused global feature vector.

[0146] The fused global feature vector X fuse is:

[0147]

[0148] The final X fuse is the result of weighted fusion of different modal features, and the model will dynamically adjust according to the relevance of different modal features at different time steps to generate a global feature representation with adaptive ability.

[0149] The multi-modal feature fusion neural network constructed by the application utilizes a multi-modal data fusion method to provide more complete grasping action representation; CNN is adopted to extract local features, and then LSTM is used to model time dependence, so that the entire system has strong expression ability in terms of feature level and time sequence level; an attention mechanism is introduced to dynamically select the most critical time or modal features for action recognition, thereby improving the robustness and performance of the model, and the model is particularly suitable for complex action sequence recognition tasks; the feature fusion suitable for diversified actions is used for weight adjustment, so that the model can adaptively adjust the weight of different modal features according to the requirements of specific actions, and this adaptive ability enables the model to automatically adjust the contribution of each modality according to different scenarios.

[0150] Step S2.5: output layer

[0151] The fused feature vector is further reduced in feature dimension through a fully connected layer, a Softmax classifier is used for classifying and recognizing the action, and the recognition label of the hand grasping action is output.

[0152] The multi-modal fusion CNN of the application can adjust the grasping action force in real time by fusing the spatial and time sequence features of muscle signals and motion state information, has closed-loop feedback capability, ensures that the key information in each modality signal is effectively utilized, and greatly improves the flexibility and stability of the grasping action. Meanwhile, the bidirectional LSTM is combined with the CNN, the time dependence of the hand action is retained, the spatial local features in each modality signal can be captured, and finally the importance of different modal features is dynamically adjusted through the attention mechanism, so that the model can automatically select the most useful features according to specific actions. The multi-modal feature fusion neural network constructed by the application has strong classification ability: through multi-modal data and time sequence modeling, the model can accurately recognize complex hand grasping actions (pre-grasping, grasping, transporting and placing, releasing and returning to the starting position).

[0153] Step S3: converting the recognition label of the grasping action into the control signal of the bionic manipulator through the BP neural network.

[0154] The traditional neural network has a receiving layer when designed, and when designing a neural network adaptive controller, the added receiving layer will bring more adaptive parameter adjustment, increase the calculation complexity of the neural network adaptive controller when adjusting parameters, and further affect the real-time control effect of the neural network adaptive controller, and the application selects a BP neural network adaptive controller.

[0155] The recognition label of the grasping action is used as the input signal of the BP neural network, and the recognition label of the grasping action is converted into the control signal of the bionic manipulator through the forward propagation of the BP neural network and then output.

[0156] The forward propagation process of the BP neural network is as follows:

[0157] h(t) = f(z(t)) = f(U h(t-1) + W u(t) + b h )

[0158] y(t) = g(s(t))

[0159] s(t) = V h(t) + b0

[0160] Wherein, u(t) represents the input signal of the BP neural network, y(t) represents the output of the output layer neuron of the BP neural network, h(t) represents the output of the hidden layer neuron of the BP neural network, z(t) represents the input of the hidden layer neuron of the BP neural network, s(t) represents the input of the output layer neuron of the BP neural network, W represents the connection weight value between the input layer and the hidden layer of the BP neural network; U represents the connection weight value between the hidden layer and the output layer of the BP neural network; V represents the connection weight value between the hidden layer and the output layer of the BP neural network, b h and b0 are the biases of the hidden layer and the output layer respectively.

[0161] The parameters of the BP neural network are adjusted online by the RTRL algorithm; the loss function L(t) of the RTRL algorithm is as follows:

[0162]

[0163] e(t) = y d (t) - y(t)

[0164] Wherein, y d (t) is the expected motion position of the manipulator, y(t) is the actual output displacement of the manipulator;

[0165] The connection weight value between the hidden layer and the output layer of the BP neural network changes as follows:

[0166] U(t+1) = U(t) + ΔU(t)

[0167]

[0168] The connection weight value between the input layer and the hidden layer of the BP neural network changes as follows:

[0169] W(t+1) = W(t) + ΔW(t)

[0170]

[0171] The connection weight value between the hidden layer and the output layer of the BP neural network changes as follows:

[0172] V(t+1) = V(t) + ΔV(t)

[0173]

[0174] Wherein, η1, η2, and η3 are the learning rates for parameter changes between the input layer and the hidden layer, the learning rates for parameter changes between the connecting layer and the hidden layer, and the learning rates for parameter changes between the output layer and the hidden layer, respectively.

[0175] The detailed derivation process is as follows:

[0176]

[0177] Where f′(x)=f(x)·(1-f(x)); g′(x)=1.

[0178] remember We can obtain:

[0179]

[0180] remember We can obtain:

[0181]

[0182] like Figures 5-11 As shown, through the position tracking of joints during movement, the angle tracking of joints during movement, the control output of joints, and the approximation results of the sGME signal by the BP neural network adaptive controller, it can be seen that the present invention significantly improves the recognition accuracy of arm grasping actions, enabling the bionic robotic hand to exhibit higher precision and stability in the recognition and execution of various complex actions.

[0183] The performance of the bionic robotic hand of this invention is evaluated using metrics such as accuracy, recall, and F1-score.

[0184]

[0185] In this system, TP represents true cases, TN represents true negative cases, FP represents false positive cases, and FN represents false negative cases. The system achieves an accuracy rate of 89.6% in recognizing the five composite actions, demonstrating good real-time performance and accuracy.

[0186] Example 2

[0187] A bionic robotic hand control system based on sEMG signals is provided, including a signal acquisition module, a motion recognition module, and a control signal generation module. The motion recognition module and the control signal generation module of this invention are deployed on a host computer, and the host computer is assembled with the bionic robotic hand. The host computer recognizes hand movements in real time, and uses a bionic hand sample with five fingers and ten degrees of freedom to control the bionic robotic hand to perform corresponding movements, thereby achieving high-precision motion imitation.

[0188] a signal collection module, configured to collect multi-modal signals; the multi-modal signals include sEMG signals and posture signals when an operator's arm is grabbing, and piezoelectric signals when a bionic robot is grabbing.

[0189] an action recognition module, configured to process the multi-modal signals by a constructed multi-modal feature fusion neural network to recognize a grabbing action; the multi-modal feature fusion neural network includes an input layer, a feature extraction layer, a time sequence modeling layer, a feature fusion layer, and an output layer; the input layer receives the multi-modal signals, the feature extraction layer extracts local time sequence features of the multi-modal signals by using a convolutional neural network, the time sequence modeling layer processes the extracted local time sequence features by using a BiLSTM network to model time sequence dependencies of the grabbing action, the feature fusion layer introduces a time-based attention mechanism to adaptively adjust feature weights of each modality, and then fuses features of the multi-modal into a global feature vector, and the output layer uses a classifier to recognize the grabbing action based on the global feature vector and outputs a recognition label of the grabbing action.

[0190] a control signal generation module, configured to convert the recognition label of the grabbing action into a control signal of the bionic robot by using a BP neural network.

[0191] For more specific processes of the above method, reference can be made to the corresponding contents disclosed in the foregoing embodiments, which will not be repeated here.

[0192] Embodiment 3

[0193] A computer device includes a processor and a memory; when the processor executes a computer program stored in the memory, the steps of the sEMG signal-based bionic robot control method according to any one of Embodiment 1 are implemented.

[0194] For more specific processes of the above method, reference can be made to the corresponding contents disclosed in the foregoing embodiments, which will not be repeated here.

[0195] Embodiment 4

[0196] A computer readable storage medium is configured to store a computer program; when the computer program is executed by a processor, the steps of the sEMG signal-based bionic robot control method according to any one of Embodiment 1 are implemented.

[0197] For more specific processes of the above method, reference can be made to the corresponding contents disclosed in the foregoing embodiments, which will not be repeated here.

[0198] The various embodiments described in the specification are progressive in nature, and each embodiment highlights the differences from other embodiments. The same or similar parts among the various embodiments can be mutually referred to. For the system, device and storage medium disclosed by the embodiments, since they correspond to the method disclosed by the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0199] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present application can be realized by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions in the embodiments of the present application can be embodied in the form of a software product, and the computer software product can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in the various embodiments or some parts of the embodiments of the present application.

[0200] The above is only the preferred embodiment of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments. Any technical solution falling within the concept of the present application shall fall within the protection scope of the present application. It should be noted that, for those skilled in the art, some improvements and refinements without departing from the principles of the present application shall be considered as the protection scope of the present application.

Claims

1. A control method of a bionic manipulator based on sEMG signals, characterized in that, The method comprises the following steps: Collecting multi-modal signals; the multi-modal signals include sEMG signals and posture signals when an operator's arm is grabbing, and piezoelectric signals when a bionic manipulator is grabbing; Processing the multi-modal signals through a constructed multi-modal feature fusion neural network to identify the grabbing action; the multi-modal feature fusion neural network comprises an input layer, a feature extraction layer, a time sequence modeling layer, a feature fusion layer, and an output layer; the input layer receives the multi-modal signals, the feature extraction layer extracts local time sequence features of the multi-modal signals by using a convolutional neural network, the time sequence modeling layer processes the extracted local time sequence features by using a BiLSTM network to model the time sequence dependency of the grabbing action; the feature fusion layer introduces a time-based attention mechanism to adaptively adjust the feature weights of each modality, and then fuses the features of the multi-modal signals into a global feature vector; the output layer uses a classifier to identify the global feature vector for the grabbing action and outputs an identification label of the grabbing action; Converting the identification label of the grabbing action into a control signal of the bionic manipulator through a BP neural network; The feature fusion layer introduces a time-based attention mechanism to adaptively adjust the feature weights of each modality, and then fuses the features of the multi-modal signals into a global feature vector, the process is as follows: The feature vectors of each modality are linearly transformed using a weight matrix learnable during training, and the linearly transformed feature vectors of each modality are spliced to obtain spliced feature vectors ; The feature vectors of each modality are linearly transformed, represented as: ； ； ； wherein, is a sEMG signal feature vector, is a corresponding learnable weight matrix; is a posture signal feature vector, is a corresponding learnable weight matrix, is a piezoelectric signal feature vector, is a corresponding learnable weight matrix; , and is a sEMG signal feature vector, a posture signal feature vector and a piezoelectric signal feature vector mapped to a unified feature space; the concatenated feature vector : ; based on the concatenated feature vectors , obtain the attention score of each time step through a nonlinear activation function ; the attention score of each time step is: wherein, is a parameter learnable in the training process; each time step attention score is converted to a normalized attention weight for each time step by a Softmax function each time step attention score is converted to a normalized attention weight for each time step by a Softmax function ; the normalized attention weight for each time step is: , wherein, is the total number of time steps; based on the obtained normalized attention weights for each time step concatenating the feature vectors for all time steps performing weighted summation to obtain the fused global feature vector The fused global feature vector is: 。 2. The sEMG signal-based biomimetic hand control method according to claim 1, characterized by, Further comprising: performing multi-layer filtering processing on the collected sEMG signals when the operator's arm is grabbing, the specific process is as follows: Preliminarily removing high-frequency noise and low-frequency noise of the sEMG signals through band-pass filtering and envelope detection; Using an EEMD decomposition method to decompose the sEMG signals after preliminary noise removal into high-frequency IMFs, low-frequency IMFs, and residual signals; Judging whether the high-frequency IMFs contain high-frequency noise, if not, directly using the high-frequency IMFs without noise to construct new high-frequency IMFs; if so, retaining the IMF components without noise, and at the same time, denoising the IMF components containing noise through wavelet thresholding, and combining the denoised IMF components with the retained IMF components without noise to construct new high-frequency IMFs; Reconstructing the new high-frequency IMFs, low-frequency IMFs, and residual signals; Performing short-time Fourier transform on the reconstructed sEMG signals to remove periodic noise; Using a Savitzky-Golay filter to smooth the sEMG signals after removing periodic noise to remove high-frequency fluctuation noise.

3. The sEMG signal-based biomimetic hand control method according to claim 1, wherein The posture signals include angular velocity signals and acceleration signals of the operator's arm; When collecting the sEMG signals, pre-processing the sEMG signals, and judging whether the current sEMG signals are a grabbing action according to the pre-processed sEMG signals, if not, directly outputting the pre-processed sEMG signals, if so, obtaining the optimal grabbing force according to the sEMG signals, and outputting the pre-processed sEMG signals. When the piezoelectric signal is collected, the piezoelectric signal is preprocessed, the current grabbing force of the mechanical hand is obtained according to the preprocessed piezoelectric signal, and it is judged whether the current grabbing force meets the value range of the optimal grabbing force; if not, the grabbing force transmitted by the piezoelectric signal is reacquired until the value range of the optimal grabbing force is met; if yes, the optimal grabbing force is output as the piezoelectric signal; When the posture signal is collected, it is judged whether the posture signal meets the preset arm action posture requirement in combination with the sEMG signal; if yes, the currently collected posture signal is output, otherwise, the currently collected posture signal is removed.

4. The sEMG signal-based biomimetic hand control method according to claim 1, wherein The recognition label of the grabbing action is converted into the control signal of the bionic mechanical hand through the BP neural network, and specifically includes: The recognition label of the grabbing action is taken as the input signal of the BP neural network, and the recognition label of the grabbing action is converted into the control signal of the bionic mechanical hand through the forward propagation of the BP neural network and is output; The forward propagation process of the BP neural network is as follows: ； ； ； wherein, represents the input signal of the BP neural network, represents the output of the output layer neuron of the BP neural network, represents the output of the hidden layer neuron of the BP neural network, represents the input of the hidden layer neuron of the BP neural network, represents the input of the output layer neuron of the BP neural network, represents the connection weight between the input layer and the hidden layer of the BP neural network; represents the connection weight between the hidden layer and the receiving layer of the BP neural network; represents the connection weight between the hidden layer and the output layer of the BP neural network, and are the hidden layer and output layer bias, respectively.

5. The sEMG signal-based biomimetic hand control method according to claim 4, characterized by, The parameters of the BP neural network are adjusted on-line by the RTRL algorithm; the loss function of the RTRL algorithm is: J = 1 2∑(y - y)2 ； ； wherein, is the desired motion position of the robot, is the actual output displacement of the robot; The connection weight value between the hidden layer and the receiving layer of the BP neural network changes as follows: ； ； The connection weight value between the input layer and the hidden layer of the BP neural network changes as follows: ； ； The connection weight value between the hidden layer and the output layer of the BP neural network changes as follows: ； ； wherein, , and are the learning rates for the parameter variations between the input and hidden layers, between the carrier and hidden layers, and between the output and hidden layers, respectively.

6. A sEMG signal-based bionic manipulator control system based on the sEMG signal-based bionic manipulator control method of any one of claims 1-5, characterized in that, It includes: The signal acquisition module is used for collecting multi-modal signals; the multi-modal signals include: sEMG signals and posture signals when the operator's arm is grabbing, and piezoelectric signals when the bionic mechanical hand is grabbing; The action recognition module is used for processing the multi-modal signals through the constructed multi-modal feature fusion neural network to identify the grabbing action; the multi-modal feature fusion neural network includes an input layer, a feature extraction layer, a time sequence modeling layer, a feature fusion layer and an output layer; the input layer receives the multi-modal signals, the feature extraction layer extracts local time sequence features of the multi-modal signals by using a convolutional neural network, the time sequence modeling layer processes the extracted local time sequence features by using a BiLSTM network to model the time sequence dependency of the grabbing action; the feature fusion layer introduces a time-based attention mechanism to adaptively adjust the feature weights of each modality, and then fuses the features of the multi-modal into a global feature vector; the output layer uses a classifier to identify the global feature vector for the grabbing action and outputs the recognition label of the grabbing action; The control signal generation module is used for converting the recognition label of the grabbing action into the control signal of the bionic mechanical hand through the BP neural network.

7. A computer device, characterized by The processor executes the computer program saved in the memory to realize the steps of the bionic mechanical hand control method based on the sEMG signal in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer program is stored in the memory and is executed by the processor to realize the steps of the bionic mechanical hand control method based on the sEMG signal in any one of claims 1-5.

Images