Hand joint angle estimation method, estimation device, storage medium and equipment

Abstract

The present invention discloses a hand joint angle estimation method, estimation device, storage medium and equipment. The hand joint angle estimation method includes: obtaining real-time surface electromyographic signals generated when the hand of the object to be measured moves; inputting the surface electromyographic signals into a pre-trained hand joint angle estimation model to obtain the estimated joint angle of the hand of the object to be measured during movement, wherein the hand joint angle estimation model includes a multi-scale convolution module and a multi-layer multi-head attention module connected in sequence. The multi-scale convolution module can better extract features related to the joint angle from the electromyographic signal time series, so compared with the existing Gaussian process regression joint angle estimation algorithm, higher regression accuracy can be achieved. In addition, the overall timing characteristics of the electromyographic sequence are learned by the multi-layer multi-head attention mechanism, which avoids the long-distance dependency problem caused by traditional recurrent neural learning, and further improves the operating efficiency through parallel computing.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular, relates to a hand joint angle estimation method, an estimation device, a computer-readable storage medium, and a computer device. Background Art

[0002] For decades, robotic hands have been widely researched and applied in fields such as search and rescue, industry, and prosthetics. Surface electromyography (EMG) signals, which precede movement and are easy to acquire, are ideal physiological signals for extracting human movement intentions. They have numerous applications in rehabilitation medicine and human-computer interaction, and are widely used worldwide for dexterous control of robotic hands. For a long time, myoelectric control often relied on discrete motion recognition to achieve robotic control. However, the hand, as the most distinctive organ in the human body, possesses a high degree of flexibility. Therefore, multi-degree-of-freedom, real-time, continuous control approaches represent the future development direction for robotic hands, providing more natural and intuitive control. To achieve continuous, real-time control strategies, many recent studies have developed algorithms that map EMG signals to finger joint angles. Furthermore, given the challenges of amputee applications and the inherent difficulty of acquiring hand EMG, arm EMG signals are often used to develop regression algorithms to estimate joint angles during hand movement. Since arm muscles activate during hand movement, estimating hand joint angles from arm EMG signals is feasible. Based on current research, there are two main approaches to estimating real-time human joint angles using electromyographic signals: model-based and data-based. Among model-based methods, the most common are physiological models such as the Hill model and the Huxley model. Model-based methods can explain the process of human movement, and the parameters in the model can express the properties of the human skeletal and muscular systems, such as muscle fiber length and tendon length. The most commonly used data-based methods are supervised methods based on machine learning and deep learning. These methods directly establish a regression algorithm from skin surface electromyographic signals to continuous motion quantities, which is simple and reliable. Currently, regression algorithms such as Gaussian process regression and long short-term memory networks have been widely used in the task of real-time estimation of hand joint angles during movement.

[0003] Although the model-based joint angle estimation method has strong interpretability, the model contains a large number of parameters that are difficult to measure directly. It is currently only used for motion estimation of several joints with low degrees of freedom. However, human hand movements require the coordination of many joints, which requires the estimation method to support more degrees of freedom. Data-based methods can often estimate relatively more degrees of freedom. The regression algorithm using Gaussian processes in existing inventions can map the skin surface electromyographic signals to multi-degree-of-freedom finger joint angles. However, during the movement process, the joint angle at the current moment may be associated with the electromyographic signals before the current moment. However, the Gaussian process regression algorithm does not take these associations into consideration. Long short-term memory networks can extract the temporal features from electromyographic signals. Most of the existing deep learning networks for estimating hand joint angles based on electromyography add long short-term memory networks to learn the overall temporal features of electromyographic sequences. Long short-term memory networks belong to recurrent neural network structures. When the sequence is very long, due to the long-range dependency problem of recurrent neural networks, part of the input sequence information is easily lost. Summary of the Invention

[0004] (1) Technical Problems to be Solved by the Present Invention

[0005] The technical problem solved by the present invention is: how to effectively extract features related to joint angles and overall timing features from electromyographic signals to improve regression accuracy and avoid the long-distance dependency problem caused by traditional recurrent neural learning.

[0006] (2) Technical solution adopted by the present invention

[0007] A method for estimating hand joint angles, comprising:

[0008] Acquire real-time surface electromyographic signals generated by the hand movements of the subject to be tested;

[0009] The surface electromyography signal is input into a pre-trained hand joint angle estimation model to obtain the estimated joint angle of the hand of the object to be measured during movement, wherein the hand joint angle estimation model includes a multi-scale convolution module and a multi-layer multi-head attention module connected in sequence.

[0010] Preferably, the method for pre-training the hand joint angle estimation model includes:

[0011] Obtaining joint angle measurements and corresponding surface electromyographic signal data during human hand movement;

[0012] Inputting the surface electromyography signal data as a training sample into the hand joint angle estimation model, and outputting the hand joint angle prediction value after being processed by the multi-scale convolution module and the multi-layer multi-head attention module in sequence;

[0013] An error loss is calculated based on the hand joint angle prediction value and the joint angle measurement value, and network parameters of the multi-scale convolution module and the multi-layer multi-head attention module are updated based on the error loss.

[0014] Preferably, the hand joint angle estimation method further includes:

[0015] Feature extraction is performed on the surface electromyography signal data to obtain a root mean square feature vector corresponding to the surface electromyography signal data, and the root mean square feature vector is used as a training sample.

[0016] Preferably, the multi-scale convolution module includes two multi-scale convolution sub-networks and a pooling layer connecting the two multi-scale convolution sub-networks, the two multi-scale convolution sub-networks have the same network structure, wherein the multi-scale convolution sub-network includes three parallel branches and a feature splicing layer, each branch includes two convolution layers, and the method for the multi-scale convolution module to process the training sample includes:

[0017] Inputting the training samples into the three branches of the first multi-scale convolutional subnetwork respectively to perform convolution feature extraction to obtain three first convolution features, and using the feature splicing layer to splice the three first convolution features to form a first convolution fusion feature;

[0018] After using the pooling layer to compress the sequence length of the first convolution fusion feature, the first convolution fusion feature is input into the three branches of the second multi-scale convolution sub-network respectively to perform convolution feature extraction to obtain three second convolution features. The three second convolution features are then spliced ​​using the feature splicing layer of the second multi-scale convolution sub-network to form a second convolution fusion feature, wherein the second convolution fusion feature is used as the input data of the multi-layer multi-head attention module.

[0019] Preferably, the hand joint angle estimation method further includes:

[0020] The second convolutional fusion feature is position-encoded and used as input data for the multi-layer multi-head attention module.

[0021] Preferably, the multi-layer multi-head attention module includes several layers of sequentially connected multi-head attention networks and a first fully connected layer, each layer of the multi-head attention network includes three single-head attention networks with the same structure and arranged in parallel, a feature splicing layer, and a second fully connected layer, wherein the method of processing the second convolutional fusion feature using the multi-layer multi-head attention module includes:

[0022] The second convolutional fusion feature after position encoding is used as the input data of the first layer multi-head attention network, and is input into the three single-head attention networks respectively, and three single-head attention output features are obtained after the first linear mapping operation;

[0023] Using the feature concatenation layer of the first-layer multi-head attention network, the three single-head attention output features are concatenated along the dimension of the feature channel to obtain an attention fusion feature;

[0024] Using the second fully connected layer to perform a second linear mapping operation on the attention fusion feature to obtain a multi-head attention output feature, the multi-head attention output feature is used as the input data of the next layer of the multi-head attention network, and the above steps are repeated until the last layer of the multi-head attention network outputs the multi-head attention output feature;

[0025] The multi-head attention output features output by the last layer of the multi-head attention network are input into the first fully connected layer to obtain a sequence of hand joint angle prediction values.

[0026] Preferably, the method of performing a first linear mapping operation on the second convolutional fusion feature after position encoding using the single-head attention network includes:

[0027] The query feature vector sequence is calculated according to the following formulas Key feature vector sequence Sum eigenvector sequence

[0028] Q=CW Q , K=CW k , V=CW v

[0029] Among them, W Q 、W k 、W v is the linear mapping operation determined by training, d q d k d v is a constant, n represents the number of signal channels of the electromyographic signal, C∈R n×l The second convolution fusion feature after representing the position encoding;

[0030] The single-head attention output feature H is obtained according to the following formula:

[0031]

[0032] The present application also discloses a hand joint angle estimation device, which includes:

[0033] A signal acquisition unit, used to acquire real-time surface electromyographic signals generated by the hand movements of the subject to be tested;

[0034] A pre-trained hand joint angle estimation model is used to predict the estimated joint angle of the hand movement of the subject to be tested based on the collected real-time surface electromyography signal, wherein the hand joint angle estimation model includes a multi-scale convolution module and a multi-layer multi-head attention module connected in sequence.

[0035] The present application also discloses a computer-readable storage medium, which stores a hand joint angle estimation program. When the hand joint angle estimation program is executed by a processor, the above-mentioned hand joint angle estimation method is implemented.

[0036] The present application also discloses a computer device, which includes a computer-readable storage medium, a processor, and a hand joint angle estimation program stored in the computer-readable storage medium. When the hand joint angle estimation program is executed by the processor, the above-mentioned hand joint angle estimation method is implemented.

[0037] (3) Beneficial effects

[0038] The present invention discloses a hand joint angle estimation method, estimation device, storage medium and equipment, which have the following technical effects compared with existing methods:

[0039] The hand joint angle estimation model used in this finger joint angle estimation method utilizes a multi-scale convolution module to better extract joint angle-related features from the EMG signal time series. This allows for higher regression accuracy than existing Gaussian process regression joint angle estimation algorithms. Furthermore, a multi-layer, multi-head attention mechanism learns the overall temporal characteristics of the EMG sequence, avoiding the long-distance dependency issues associated with traditional recurrent neural network learning. Parallel computing further improves operational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flowchart of a hand joint angle estimation method according to a first embodiment of the present invention;

[0041] Figure 2 This is a training flow chart of the hand joint angle estimation model according to the first embodiment of the present invention;

[0042] Figure 3 This is a network structure diagram of the hand joint angle estimation model according to the first embodiment of the present invention;

[0043] Figure 4 A comparison diagram of the joint angle estimation values ​​and actual joint angles obtained by the hand joint angle estimation model according to the first embodiment of the present invention;

[0044] Figure 5 This is a functional block diagram of a hand joint angle estimation device according to a second embodiment of the present invention;

[0045] Figure 6 Schematic diagram of a computer device according to a fourth embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0047] Before describing the various embodiments of the present application in detail, the inventive concept of the present application is first briefly described: the data-based deep learning models in the prior art are mainly based on the regression algorithm based on the Gaussian process and the long short-term memory network. The former does not fully consider the correlation between the joint angle and the electromyographic signal, resulting in low regression accuracy. The latter is prone to losing part of the input sequence information when facing a long sequence due to the existence of long-range dependency problems. To this end, the hand joint angle estimation method provided in this embodiment utilizes a multi-scale convolution module and a multi-layer multi-head attention module to construct a hand joint angle estimation model, and trains the hand joint angle estimation model, so that the hand joint angle estimation model fully learns the correlation between the joint angle and the surface electromyographic signal and the overall timing characteristics of the surface electromyographic signal, which is beneficial to improving prediction accuracy and avoiding the long-range dependency problem of traditional recurrent neural learning.

[0048] Specifically, if Figure 1 As shown, the hand joint angle estimation method of the first embodiment includes the following steps:

[0049] Step S10: acquiring real-time surface electromyographic signals generated by the hand movement of the subject to be measured;

[0050] Step S20: inputting the surface electromyography signal into a pre-trained hand joint angle estimation model to obtain the estimated joint angle of the hand of the subject to be measured during movement.

[0051] Among them, the hand joint angle estimation model includes multi-scale convolution modules and multi-layer multi-head attention modules connected in sequence, such as Figure 2 As shown, the method for pre-training the hand joint angle estimation model includes the following steps:

[0052] Step S101: obtaining joint angle measurement values ​​and corresponding surface electromyography signal data of a human hand during movement;

[0053] Step S102: inputting the surface electromyography signal data as a training sample into the hand joint angle estimation model, and outputting a hand joint angle prediction value after being processed by the multi-scale convolution module and the multi-layer multi-head attention module in sequence;

[0054] Step S103: Calculate the error loss based on the hand joint angle prediction value and the joint angle measurement value, and update the network parameters of the multi-scale convolution module and the multi-layer multi-head attention module based on the error loss.

[0055] For example, in step S101, the myoelectric differential electrode of the Delsys myoelectric acquisition system is used as the myoelectric sensor to collect surface electromyographic signals. The collected surface electromyographic signals come from the extensor digitorum, flexor digitorum, biceps brachii, triceps brachii, and a circle of muscles 2 to 6 cm away from the elbow of the forearm. The myoelectric signal sampling frequency is 2000 Hz. A 5 to 450 Hz Butterworth filter is used to bandpass filter the myoelectric signal for baseline correction and noise removal of the myoelectric signal. A data glove of model CyberGlove II is used to collect hand joint angle data with a sampling frequency of 20 Hz. For the joint angle, it is first resampled to 2000 Hz to ensure that the surface electromyographic signal and the joint angle sequence are synchronized in time. Then, a 2 Hz zero-phase low-pass filter is used to smooth the original joint angle signal to avoid step jitter in the signal and make it more like the normal movement curve of the human body. The maximum and minimum values ​​of the collected electromyographic data and joint angle data are further recorded for normalization of training and test data.

[0056] Furthermore, this embodiment also performs feature extraction on the surface electromyography signal data to obtain the root mean square feature vector corresponding to the surface electromyography signal data, and uses the root mean square feature vector as the input to the hand joint angle estimation model. Specifically, with a sliding window length of 100 sampling points, after each calculation of the root mean square feature of the current window is completed, it slides backward by 1 sampling point. This can generate a feature sequence that is the same length as the surface electromyography signal of the original skin and corresponds to the time point. This method can maximize the amount of data. Finally, a sliding window is used to generate a root mean square feature vector sequence and a joint angle sequence with a window length of 400 sampling points. The sliding window step size is 100 sampling points. The joint angle and root mean square feature data in each sliding window will be used as a sample data, and the joint angle vector dimension represents the estimated number of joint angles. In this embodiment, 60% of the data samples are used as training data, and 40% of the data samples are used as test data.

[0057] For example, in step S102, Figure 3As shown, the multi-scale convolution module includes two multi-scale convolution sub-networks and a pooling layer connecting the two multi-scale convolution sub-networks. The first dotted box and the second dotted box represent the first multi-scale convolution sub-network and the second multi-scale convolution sub-network, respectively. The two multi-scale convolution sub-networks have the same network structure, wherein the multi-scale convolution sub-network includes three parallel branches and a feature splicing layer, and each branch includes two convolution layers. Among them, the first convolution layer of each branch is a one-dimensional convolution layer with a convolution kernel size of 1, and the second convolution layer of each branch is also a one-dimensional convolution layer. The convolution kernel size of the second convolution layer of each branch is different, and the convolution kernel size ranges from 3 to 11, which needs to be selected according to the experimental situation. The feature splicing layer is used to splice the sequences obtained from the three branches along the feature direction, and the dimension range of the spliced ​​output features is 128 to 512. The pooling layer is used to compress the sequence length of the output features of the first multi-scale convolutional sub-network. In this embodiment, the pooling layer compresses the sequence length of the output features to half of the original length. The dimension of the output features of the second multi-scale convolutional sub-network is 32 to 128. Specifically, in step S102, the method for the multi-scale convolution module to process the training sample includes the following steps:

[0058] Step S1021: input the training samples into the three branches of the first multi-scale convolutional subnetwork respectively to perform convolution feature extraction to obtain three first convolution features, and use the feature splicing layer to splice the three first convolution features to form a first convolution fusion feature.

[0059] Step S1022: Use the pooling layer to compress the sequence length of the first convolutional fusion feature and input it into the three branches of the second multi-scale convolutional sub-network respectively to perform convolutional feature extraction to obtain three second convolutional features. Then use the feature splicing layer of the second multi-scale convolutional sub-network to splice the three second convolutional features to form a second convolutional fusion feature, where the second convolutional fusion feature serves as the input data of the multi-layer multi-head attention module.

[0060] As a preferred embodiment, after the convolution operation is performed on the above two multi-scale convolutional sub-networks, a zero-padding operation is performed to ensure that the input and output sequences are of equal length, and an exponential linear unit activation function is added after the above two multi-scale convolutional sub-networks to increase the nonlinearity of the deep learning network.

[0061] Furthermore, to allow the attention mechanism to distinguish the temporal order of EMG signals, this embodiment also uses additional position encoding, namely, position encoding of the second convolutional fusion feature. Specifically, this embodiment adds the second convolutional fusion feature to the absolute position encoding before feeding it into the multi-layer multi-head attention mechanism module. The position encoding used in this embodiment adopts sinusoidal absolute position encoding, which is a relatively common position encoding method. The specific position encoding process is not detailed here.

[0062] Specifically, in step S102, the multi-layer multi-head attention module includes several layers of multi-head attention networks connected in sequence and a first fully connected layer. The multi-head attention network is as follows: Figure 3 As shown in the third dashed box, each layer of the multi-head attention network consists of three single-head attention networks with identical structures arranged in parallel, a feature concatenation layer, and a second fully connected layer. The number of layers in the multi-head attention network ranges from 2 to 4, and the specific number of layers is selected based on actual conditions.

[0063] The method of processing the second convolutional fusion feature using a multi-layer multi-head attention module includes the following steps:

[0064] Step S1023: The second convolutional fusion features after position encoding are used as input data of the first layer multi-head attention network, and are input into the three single-head attention networks respectively, and three single-head attention output features are obtained after the first linear mapping operation.

[0065] Specifically, the query feature vector sequence is calculated according to the following formulas: Key feature vector sequence Sum eigenvector sequence

[0066] Q=CW Q , K=CW k , V=CW v

[0067] Among them, W Q 、W k 、W v is the linear mapping operation determined by training, d q d k d v It is a constant, generally set to 32~128, and needs to be adjusted according to actual conditions. n×l represents the second convolutional fusion feature after position encoding, and n represents the number of signal channels of the electromyographic signal.

[0068] The single-head attention output feature H is obtained according to the following formula:

[0069]

[0070] Among them, d k It is a manually set parameter used to scale the value to prevent the calculated result from being too large and causing numerical instability.

[0071] Step S1024: Use the feature concatenation layer of the first-layer multi-head attention network to concatenate the three single-head attention output features H along the dimension of the feature channel to obtain the attention fusion feature.

[0072] Step S1025: Use the second fully connected layer to perform a second linear mapping operation on the attention fusion feature to obtain a multi-head attention output feature. The multi-head attention output feature is used as the input data of the next layer of the multi-head attention network. Repeat the above steps until the last layer of the multi-head attention network outputs the multi-head attention output feature. It should be noted that repeating the above steps described here refers to repeating steps S1023 to S1025.

[0073] The operation of the feature concatenation layer and the second fully connected layer of the multi-head attention network at each layer can be expressed as follows:

[0074] Z=Concat(H1,H2,H3)W O

[0075] Among them, Z represents the multi-head attention output feature, H1, H2, H3 represent the three single-head attention output features, Concat represents the feature splicing operation, and W O Represents the second linear mapping operation that needs to be determined through training.

[0076] Step S1026: Input the multi-head attention output features output by the last layer of the multi-head attention network into the first fully connected layer to obtain a sequence of hand joint angle prediction values.

[0077] After being processed by several layers of the multi-head attention network, the first fully connected layer processes the multi-head attention output features of the last layer of the multi-head attention network to generate a sequence of hand joint angle prediction values. For example, the last vector of the hand joint angle prediction value sequence is used as the hand joint angle prediction value.

[0078] In step S103, the mean square error loss is calculated based on the hand joint angle prediction value and the joint angle measurement value, and the network parameters of the multi-scale convolution module and the multi-layer multi-head attention module are updated by the mean square error loss through an adaptive matrix estimation algorithm. The relevant calculation process and update process are existing technologies and will not be described in detail in this application.

[0079] After the above training, a hand joint angle estimation model with optimal parameters is obtained. In order to evaluate the performance of the model, the test data set is input into the hand joint angle estimation model to obtain a continuously estimated joint angle curve, which is compared with the actual joint angle curve obtained by the joint angle sensor. Furthermore, the Pearson correlation coefficient (CC) and root mean square error (RMSE) are used as evaluation criteria for the model. The Pearson correlation coefficient (CC) is calculated as follows:

[0080]

[0081] Among them, θ est 、 θ real and They represent the predicted joint angle, the average of the predicted joint angle, the actual joint angle, and the average of the actual joint angle. The root mean square error (RMSE) is calculated as follows:

[0082]

[0083] During the experiment of this embodiment, the joint angle data and electromyographic data of 15 subjects when completing the grasping action of objects of different sizes were used to train and test the hand joint angle estimation model. 60% of the data of each subject was used for model training, and the other 40% was used for model testing. The RMSE and CC regression model performance indicators of the proximal interphalangeal joints and metacarpophalangeal joints of the hand were tested. As shown in Table 1, the test results show that both indicators are greater than the two commonly used joint angle regression algorithms (algorithms based on long short-term memory networks and algorithms based on sparse pseudo-input Gaussian processes). This shows that the hand joint angle estimation model adopted in this embodiment can better achieve the joint angle estimation task. Among them, the estimation effect of this embodiment on some joints is as follows Figure 4 shown. Figure 4 The effect of using the hand joint angle estimation method of this embodiment to estimate the finger joint angle at each moment is demonstrated, where the solid line represents the true value of the finger joint angle, and the dotted line represents the predicted value of the method of this embodiment. The experimental results show that the predicted value (dashed line) of the finger joint angle by the method of this embodiment is similar to the actual measured value (solid line) and there is no distortion. For the proximal joints of the five fingers that often move, this method predicts the accurate movement trend and can match the real movement pattern of the finger joints. When the joint angle changes significantly, the prediction of this algorithm remains stable without distortion.

[0084] Evaluation indicators Model 1 of this embodiment Long Short-Term Memory Network Sparse Pseudo-Input Gaussian Process CC 0.87±0.01 0.79±0.01 0.75±0.01 RMSE 9.65±0.55 11.67±0.69 12.07±0.75

[0085] Table 1 Regression performance evaluation of three different algorithms in the same hand multi-joint angle estimation task

[0086] In summary, the finger joint angle estimation method provided in Example 1, through the multi-scale convolution module, can better extract features related to joint angles from the EMG signal time series. Therefore, compared with the existing Gaussian process regression joint angle estimation algorithm, it can achieve higher regression accuracy. At the same time, the overall timing characteristics of the EMG sequence are learned through a multi-layer multi-head attention mechanism, avoiding the long-distance dependency problem caused by traditional recurrent neural learning, and further improving operational efficiency through parallel computing.

[0087] like Figure 5 As shown, embodiment 2 also discloses a hand joint angle estimation device, which includes a signal acquisition unit 100 and a pre-trained hand joint angle estimation model 200. The signal acquisition unit 100 is used to collect real-time surface electromyography signals generated when the hand of the object to be measured moves; the pre-trained hand joint angle estimation model 200 is used to predict the estimated joint angle of the hand of the object to be measured when it moves based on the collected real-time surface electromyography signals, wherein the hand joint angle estimation model 200 includes a multi-scale convolution module and a multi-layer multi-head attention module connected in sequence. The specific working process of the signal acquisition unit 100 and the hand joint angle estimation model 200 can be found in the description of embodiment 1 and will not be repeated here.

[0088] The third embodiment further discloses a computer-readable storage medium, which stores a hand joint angle estimation program. When the hand joint angle estimation program is executed by a processor, the above-mentioned hand joint angle estimation method is implemented.

[0089] Furthermore, the fourth embodiment also discloses a computer device, at the hardware level, such as Figure 6 As shown, the computer device includes a processor 12, an internal bus 13, a network interface 14, and a computer-readable storage medium 11. The processor 12 reads the corresponding computer program from the computer-readable storage medium and then runs it, forming a request processing device at the logical level. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices. A hand joint angle estimation program is stored on the computer-readable storage medium 11, and the hand joint angle estimation program implements the above-mentioned hand joint angle estimation method when executed by the processor.

[0090] Computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0091] The above describes in detail the specific implementation methods of the present invention. Although some embodiments have been shown and described, those skilled in the art should understand that these embodiments can be modified and improved without departing from the principles and spirit of the present invention, the scope of which is defined by the claims and their equivalents. These modifications and improvements should also be within the scope of protection of the present invention.

Claims

1. A hand joint angle estimation method, characterized in that: The hand joint angle estimation method comprises: Acquire real-time surface electromyographic signals generated by the hand movements of the subject to be tested; Inputting the surface electromyography signal into a pre-trained hand joint angle estimation model to obtain an estimated joint angle of the hand of the subject to be measured during movement, wherein the hand joint angle estimation model includes a multi-scale convolution module and a multi-layer multi-head attention module connected in sequence; The multi-scale convolution module includes two multi-scale convolution sub-networks and a pooling layer connecting the two multi-scale convolution sub-networks. The network structures of the two multi-scale convolution sub-networks are the same, wherein the multi-scale convolution sub-networks include three parallel branches and a feature splicing layer, and each branch includes two convolution layers; the multi-layer multi-head attention module includes several layers of multi-head attention networks connected in sequence and a first fully connected layer. Each layer of the multi-head attention network includes three single-head attention networks with the same structure and arranged in parallel, a feature splicing layer, and a second fully connected layer.

2. The hand joint angle estimation method according to claim 1, characterized in that: The method for pre-training the hand joint angle estimation model includes: Obtaining joint angle measurements and corresponding surface electromyographic signal data during human hand movement; Inputting the surface electromyography signal data as a training sample into the hand joint angle estimation model, and outputting the hand joint angle prediction value after being processed by the multi-scale convolution module and the multi-layer multi-head attention module in sequence; An error loss is calculated based on the hand joint angle prediction value and the joint angle measurement value, and network parameters of the multi-scale convolution module and the multi-layer multi-head attention module are updated based on the error loss.

3. The hand joint angle estimation method according to claim 2, characterized in that: The hand joint angle estimation method further includes: Feature extraction is performed on the surface electromyography signal data to obtain a root mean square feature vector corresponding to the surface electromyography signal data, and the root mean square feature vector is used as a training sample.

4. The hand joint angle estimation method according to claim 3, characterized in that: The method for the multi-scale convolution module to process the training sample includes: Inputting the training samples into the three branches of the first multi-scale convolutional subnetwork respectively to perform convolution feature extraction to obtain three first convolution features, and using the feature splicing layer to splice the three first convolution features to form a first convolution fusion feature; After using the pooling layer to compress the sequence length of the first convolution fusion feature, the first convolution fusion feature is input into the three branches of the second multi-scale convolution sub-network respectively to perform convolution feature extraction to obtain three second convolution features. The three second convolution features are then spliced ​​using the feature splicing layer of the second multi-scale convolution sub-network to form a second convolution fusion feature, wherein the second convolution fusion feature is used as the input data of the multi-layer multi-head attention module.

5. The hand joint angle estimation method according to claim 4, characterized in that: The hand joint angle estimation method further includes: The second convolutional fusion feature is position-encoded and used as input data for the multi-layer multi-head attention module.

6. The hand joint angle estimation method according to claim 5, characterized in that: The method for processing the second convolutional fusion feature using the multi-layer multi-head attention module includes: The second convolutional fusion feature after position encoding is used as the input data of the first layer multi-head attention network, and is input into the three single-head attention networks respectively, and three single-head attention output features are obtained after the first linear mapping operation; Using the feature concatenation layer of the first-layer multi-head attention network, the three single-head attention output features are concatenated along the dimension of the feature channel to obtain an attention fusion feature; Using the second fully connected layer to perform a second linear mapping operation on the attention fusion feature to obtain a multi-head attention output feature, the multi-head attention output feature is used as the input data of the next layer of the multi-head attention network, and the above steps are repeated until the last layer of the multi-head attention network outputs the multi-head attention output feature; The multi-head attention output features output by the last layer of the multi-head attention network are input into the first fully connected layer to obtain a sequence of hand joint angle prediction values.

7. The hand joint angle estimation method according to claim 6, characterized in that: The method of performing a first linear mapping operation on the second convolutional fusion feature after position encoding using the single-head attention network includes: The query feature vector sequence is calculated according to the following formulas Key feature vector sequence Sum eigenvector sequence Q=CW Q ,K=CW k ,V=CW v Among them, W Q 、W k 、W v is the linear mapping operation determined by training, d q d k d v is a constant, n represents the number of signal channels of the electromyographic signal, C∈R n×l The second convolution fusion feature after representing the position encoding; The single-head attention output feature H is obtained according to the following formula:

8. A hand joint angle estimation device, characterized in that: The hand joint angle estimation device comprises: A signal acquisition unit, used to acquire real-time surface electromyographic signals generated by the hand movements of the subject to be tested; A pre-trained hand joint angle estimation model is used to predict the estimated joint angles of the hand of the subject to be measured during movement based on the collected real-time surface electromyography signals, wherein the hand joint angle estimation model includes a multi-scale convolution module and a multi-layer multi-head attention module connected in sequence; The multi-scale convolution module includes two multi-scale convolution sub-networks and a pooling layer connecting the two multi-scale convolution sub-networks. The network structures of the two multi-scale convolution sub-networks are the same, wherein the multi-scale convolution sub-networks include three parallel branches and a feature splicing layer, and each branch includes two convolution layers; the multi-layer multi-head attention module includes several layers of multi-head attention networks connected in sequence and a first fully connected layer. Each layer of the multi-head attention network includes three single-head attention networks with the same structure and arranged in parallel, a feature splicing layer, and a second fully connected layer.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a hand joint angle estimation program, and when the hand joint angle estimation program is executed by a processor, the hand joint angle estimation method according to any one of claims 1 to 7 is implemented.

10. A computer device, characterized in that: The computer device includes a computer-readable storage medium, a processor, and a hand joint angle estimation program stored in the computer-readable storage medium. When the hand joint angle estimation program is executed by the processor, it implements the hand joint angle estimation method described in any one of claims 1 to 7.

Images