Myoelectricity vision multi-feature fusion gesture recognition method and device

Abstract

The invention discloses a gesture recognition method and device based on myoelectricity vision multi-feature fusion. The method comprises the steps that a vision signal and a myoelectricity signal are collected; obtaining a first joint distance feature and a first dual-scale motion feature; obtaining a first electromyographic signal original feature and a first electromyographic signal change feature; respectively inputting the first joint distance feature, the first dual-scale motion feature, the first electromyographic signal original feature and the first electromyographic signal change feature into an embedding layer to obtain a second joint distance feature, a second dual-scale motion feature, a second electromyographic signal original feature and a second electromyographic signal change feature; inputting the second joint distance feature, the second dual-scale motion feature, the second electromyographic signal original feature and the second electromyographic signal change feature into a fusion module to obtain a channel attention map; and obtaining a gesture recognition result according to the channel attention map. The gesture recognition efficiency and accuracy can be improved, and the method can be widely applied to the technical field of computers.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for gesture recognition by fusion of myoelectric vision and multiple features. Background Art

[0002] Gestures are a natural way of communication in human society. Gesture recognition is an important interface for human-computer interaction. Establishing gesture recognition systems in the automotive field, home automation systems, and multimedia applications can significantly improve user experience. In addition, grasping operations based on gesture recognition are an important direction for real-world robotics applications, but the exploration of this field is still insufficient.

[0003] Most of the existing gesture recognition research is single-modal recognition. For example, only using image information for recognition, this vision-based method is low-cost and easy to deploy, but sensitive to complex backgrounds (such as occlusion problems); only using sensor glove data for recognition, the data accuracy is high but the equipment cost is high and data collection is difficult; only using surface electromyography (sEMG) for gesture recognition can capture the temporal information or spatial information of the wrist, but this method is easily affected by signal noise. Summary of the invention

[0004] In view of this, the main purpose of the embodiments of the present invention is to provide a gesture recognition method and device with myoelectric vision multi-feature fusion, in order to solve at least one of the problems in the prior art and to improve the efficiency and accuracy of gesture recognition.

[0005] To achieve the above object, an embodiment of the present invention provides a gesture recognition method based on myoelectric vision multi-feature fusion, the method comprising:

[0006] Collect visual signals and electromyographic signals;

[0007] Acquire a first joint distance feature and a first dual-scale motion feature according to the visual signal;

[0008] According to the electromyographic signal, obtaining the original feature of the first electromyographic signal and the change feature of the first electromyographic signal;

[0009] Input the first joint distance feature, the first dual-scale motion feature, the first electromyographic signal original feature, and the first electromyographic signal change feature into the embedding layer respectively to obtain a second joint distance feature, a second dual-scale motion feature, a second electromyographic signal original feature, and a second electromyographic signal change feature;

[0010] Input the second joint distance feature, the second dual-scale motion feature, the second electromyographic signal original feature, and the second electromyographic signal change feature into a fusion module to obtain a target channel attention map;

[0011] According to the target channel attention map, a gesture recognition result is obtained.

[0012] In some embodiments, collecting visual signals and electromyographic signals comprises the following steps:

[0013] Capturing RGB video through a first device;

[0014] Performing posture estimation on the RGB video to obtain the visual signal;

[0015] The electromyographic signal is collected by a second device.

[0016] In some embodiments, acquiring the first joint distance feature and the first dual-scale motion feature according to the visual signal comprises the following steps:

[0017] Obtaining the Euclidean distance between any pair of skeleton key points in the visual signal and constructing a symmetric matrix;

[0018] Obtaining the first joint distance feature according to the lower triangular matrix of the symmetric matrix;

[0019] Acquire the first dual-scale motion feature according to the frame number and the joint coordinate set corresponding to the frame number;

[0020] The first dual-scale motion feature includes a first slow motion feature and a first fast motion feature.

[0021] In some embodiments, the step of obtaining the first electromyographic signal original feature and the first electromyographic signal change feature according to the electromyographic signal comprises the following steps:

[0022] According to the electromyographic signal, obtaining the original feature of the first electromyographic signal;

[0023] According to the original features of the first electromyographic signal of the current frame and the original features of the first electromyographic signal of the next frame, the change features of the first electromyographic signal of the current frame are obtained.

[0024] In some embodiments, the formula used to obtain the first electromyographic signal change feature of the current frame according to the first electromyographic signal original feature of the current frame and the first electromyographic signal original feature of the next frame includes:

[0025] SVF t =o t+1 -o t,t∈1,2,…,T-1

[0026] Among them, SVF t Represents the first electromyographic signal change feature of the current frame; t Represents the original feature of the first electromyographic signal of the current frame; t+1 Represents the original feature of the first electromyographic signal of the next frame; t represents the tth frame; T represents the total number of frames.

[0027] In some embodiments, the first joint distance feature, the first dual-scale motion feature, the first electromyographic signal original feature, and the first electromyographic signal change feature are respectively input into the embedding layer to obtain the second joint distance feature, the second dual-scale motion feature, the second electromyographic signal original feature, and the second electromyographic signal change feature, including the following steps:

[0028] The first joint distance feature, the first dual-scale motion feature, the first electromyographic signal original feature and the first electromyographic signal change feature are respectively embedded into the potential vector of each frame, and the embedded features corresponding to the first joint distance feature, the first dual-scale motion feature, the first electromyographic signal original feature and the first electromyographic signal change feature are obtained respectively, and the second joint distance feature, the second dual-scale motion feature, the second electromyographic signal original feature and the second electromyographic signal change feature are obtained.

[0029] In some embodiments, the step of inputting the second joint distance feature, the second dual-scale motion feature, the second electromyographic signal original feature, and the second electromyographic signal change feature into a fusion module to obtain a target channel attention map comprises the following steps:

[0030] Connecting the second joint distance feature, the second dual-scale motion feature, the second electromyographic signal original feature, and the second electromyographic signal change feature in the channel dimension to obtain a total feature;

[0031] The total features are processed through a global average pooling layer, a fully connected layer, and a sigmoid function to generate an initial channel attention map;

[0032] In the channel dimension, the weight ratio of the initial channel attention map in each spatial dimension is adjusted to obtain an intermediate channel attention map;

[0033] The intermediate channel attention map is added to the total feature to obtain the target channel attention map.

[0034] In some embodiments, obtaining a gesture recognition result according to the target channel attention map includes the following steps:

[0035] The target channel attention map is convolved and pooled through a temporal convolution block to obtain an intermediate feature map;

[0036] The intermediate feature map is passed through a fully connected layer to obtain the gesture recognition result.

[0037] To achieve the above object, another aspect of an embodiment of the present invention provides a gesture recognition device for fusion of electromyography and vision multiple features, the device comprising:

[0038] The first module is used to collect visual signals and electromyographic signals;

[0039] A second module is used to obtain a first joint distance feature and a first dual-scale motion feature according to the visual signal;

[0040] The third module is used to obtain the original characteristics of the first electromyographic signal and the change characteristics of the first electromyographic signal according to the electromyographic signal;

[0041] A fourth module is used to input the first joint distance feature, the first dual-scale motion feature, the first electromyographic signal original feature, and the first electromyographic signal change feature into an embedding layer respectively to obtain a second joint distance feature, a second dual-scale motion feature, a second electromyographic signal original feature, and a second electromyographic signal change feature;

[0042] A fifth module is used to input the second joint distance feature, the second dual-scale motion feature, the second electromyographic signal original feature and the second electromyographic signal change feature into a fusion module to obtain a target channel attention map;

[0043] The sixth module is used to obtain a gesture recognition result according to the target channel attention map.

[0044] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the gesture recognition method of electromyography-vision multi-feature fusion described above.

[0045] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the gesture recognition method of electromyography-vision multi-feature fusion described above.

[0046] To achieve the above object, another aspect of the embodiment of the present invention provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the aforementioned gesture recognition method of electromyography-vision multi-feature fusion.

[0047] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a gesture recognition method and device with myoelectric visual multi-feature fusion. The scheme collects visual signals and electromyographic signals, and can convert the information of these two modalities into four transformation features, including a first joint distance feature, a first dual-scale motion feature, a first electromyographic signal original feature, and a first electromyographic signal change feature. These four transformation features are then used for processing, and feature fusion based on channel attention is performed, thereby enhancing the model's ability to understand gestures and improving the accuracy and speed of gesture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 It is a flow chart of a gesture recognition method of myoelectric vision multi-feature fusion provided by an embodiment of the present invention;

[0050] Figure 2 is a diagram of a gesture recognition network structure of multi-feature fusion provided by an embodiment of the present invention;

[0051] Figure 3 It is a schematic diagram of a robot grasping remote control system for gesture recognition with multi-feature fusion provided by an embodiment of the present invention;

[0052] Figure 4 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention, and they are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0054] It should be noted that, although the functional modules are divided in the system schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the flow chart. The terms "first / S100" and "second / S200" in the specification and claims and the above-mentioned drawings may be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiment of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0055] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0057] Gestures are a widely used communication method in social communication. People use gestures to express various information. Gesture recognition refers to the process of identifying and interpreting human hand movements or postures through computer systems. Gesture recognition combines pattern recognition technology with advanced perception technology, involving multiple disciplines such as computer graphics, mathematics, medicine, physiology, and artificial intelligence. It is a multidisciplinary research topic and an important part of achieving natural human-computer interaction. Gesture recognition technology stands out among new human-computer interaction technologies and is expected to become the mainstream human-computer interaction channel after the mouse, keyboard, and touch screen.

[0058] According to the movement characteristics of gestures, gestures can be divided into static gestures and dynamic gestures. Static gestures refer to gestures in which the shape and position of the hand remain unchanged and information is conveyed through the outline of the hand. Dynamic gestures refer to gestures in which the overall position and joint position of the hand change, thus containing richer information. Such gestures require the collection of sequence data for processing. Research on gesture recognition technology has both theoretical significance and broad application prospects.

[0059] Gesture recognition technology can be divided into two categories: gesture recognition technology based on visual images, which mainly relies on cameras for gesture recognition and tracking; and technology based on wearable devices, which obtains hand gesture information for recognition through various devices worn by users on their hands or arms, such as gloves and bracelets.

[0060] If gesture recognition is performed using only single-modal data, the recognition results will be bottlenecked due to some defects in the data itself. Multimodal methods integrate information from multiple modalities, which effectively improves recognition performance compared to single-modal methods. Therefore, in recent years, many works have focused on studying multimodal gesture recognition methods. Existing methods consider using surface electromyography and camera-based visual input for complementary fusion, directly merging the reasoning decisions of each sensor at the classifier level. However, these late fusion methods ignore the potential correlation between modalities during feature extraction, and require multiple manual adjustments based on the results of each modality to avoid reducing the overall recognition rate due to poor performance of one modality. Frequent manual intervention is required to adjust parameters, which increases the complexity and cost of the operation and leads to low operating efficiency of the system. In the prior art, there is also a method that uses data-level fusion to combine the RGB image, depth image and sEMG signal of the gesture into a consistent structure, and then inputs them into the network for final gesture recognition. This method solves the problem of gesture recognition under hand occlusion and enhances the reliability and security of gesture-based HRI. However, combining features before the feature extraction stage may produce high-dimensional feature vectors, which increases computational complexity and storage requirements. In addition, noise in the original data may be directly transmitted to the subsequent network, affecting the recognition performance.

[0061] In view of this, if Figure 1 As shown, the embodiment of the present invention provides a gesture recognition method based on myoelectric vision multi-feature fusion, which may include but is not limited to steps S100 to S600:

[0062] Step S100, collecting visual signals and electromyographic signals;

[0063] Step S200, acquiring a first joint distance feature and a first dual-scale motion feature according to the visual signal;

[0064] Step S300, obtaining a first electromyographic signal original feature and a first electromyographic signal change feature according to the electromyographic signal;

[0065] Step S400, inputting the first joint distance feature, the first dual-scale motion feature, the first electromyographic signal original feature, and the first electromyographic signal change feature into an embedding layer respectively to obtain a second joint distance feature, a second dual-scale motion feature, a second electromyographic signal original feature, and a second electromyographic signal change feature;

[0066] Step S500, inputting the second joint distance feature, the second dual-scale motion feature, the second electromyographic signal original feature, and the second electromyographic signal change feature into a fusion module to obtain a target channel attention map;

[0067] Step S600, obtaining a gesture recognition result according to the target channel attention map.

[0068] In some embodiments, reference Figure 2 In order to understand complex and practical gestures and better integrate gesture recognition technology into human-computer interaction systems, a lightweight network based on RGB video and (Surface Electromyography, sEMG) surface electromyography signals is designed. At the same time, the lightweight network uses a feature-level fusion module to enhance the fusion of multi-modal / multi-features to improve the accuracy and real-time performance of gesture recognition. Due to the different data forms of RGB and sEMG signals, the skeleton visual signal is obtained by using a posture estimation algorithm, thereby achieving efficient feature-level fusion with the sEMG signal. For example, Figure 2 As shown, a dual-branch framework based on (surface) electromyographic signals and visual signals is constructed in the lightweight network (since the skeleton data is extracted from the RGB video, the skeleton data can be used as the visual signal); the joint distance features and dual-scale motion features formed in the visual signal branch, as well as the electromyographic signal original features and electromyographic signal change features formed in the electromyographic signal branch, are all set as multi-feature inputs to extract various information related to the gesture.

[0069] In step S100 of some embodiments, multimodal / multi-feature information can be obtained by collecting visual signals and electromyographic signals. Using the electromyographic signal data as an information supplement to the visual signal data can avoid information loss caused by a single modality, help provide more detailed gesture understanding, and achieve higher-precision gesture recognition.

[0070] In some embodiments, step S100 may include but is not limited to steps S110 to S130:

[0071] Step S110, collecting RGB video through the first device;

[0072] Step S120, performing posture estimation on the RGB video to obtain the visual signal;

[0073] Step S130: collecting the electromyographic signal through a second device.

[0074] In step S110 of some embodiments, the RGB video is acquired by capturing through a first device, which may be a video camera or a webcam, or a device equipped with a video camera or a webcam, or a device equipped with a video capture card or screen recording software, which is not limited here.

[0075] In step S120 of some embodiments, the collected RGB video is converted into skeleton data (i.e., visual signal), and the skeleton data can be obtained by performing posture estimation on the RGB video. Exemplarily, the RGB video is subjected to posture estimation processing by the real-time hand posture and key point recognition algorithm mediapipe hands, and the 21 skeleton points of the human hand are recognized in real time, and the skeleton data can be obtained in the format of [64, 21, 2], where 64 represents a total of 64 frames of the video, 21 represents 21 skeleton key points, and 2 represents that each key point has two dimensions of xy. Since the data forms of RGB video data and electromyographic signals are different, the skeleton data is obtained by using a posture estimation algorithm, which lays a good foundation for the subsequent efficient feature-level fusion of skeleton data and electromyographic signal data.

[0076] In step S130 of some embodiments, the electromyographic signal is acquired by a second device, and the electromyographic signal and the RGB video are acquired simultaneously. The electromyographic signal and the RGB video have the same number of frames, and the format of the electromyographic signal is [64, 8], where 64 indicates 64 frames and 8 indicates that each frame has 8 level data. The second device may be an eight-channel MYO armband.

[0077] In step S200 of some embodiments, two transformation features, joint distance feature + dual-scale motion feature, are formed in the visual signal branch of the multimodal gesture recognition network to describe gestures from the perspective of visual invariance and visual global motion, respectively. By combining joint distance features and dual-scale motion features, gestures can be described more comprehensively. This feature combination can adapt to different gesture changes, including changes in speed and amplitude, making the recognition system more robust, thereby improving the accuracy of gesture recognition.

[0078] In some embodiments, step S200 may include but is not limited to steps S210 to S230:

[0079] Step S210, obtaining the Euclidean distance between any pair of skeleton key points in the visual signal, and constructing a symmetric matrix;

[0080] Step S220, obtaining the first joint distance feature according to the lower triangular matrix of the symmetric matrix;

[0081] Step S230, acquiring the first dual-scale motion feature according to the frame number and the joint coordinate set corresponding to the frame number;

[0082] In steps S210 to S220 of some embodiments, the gesture is described from the perspective of visual invariance, and the first joint distance feature formed contains the invariant position information of the skeleton sequence data, which helps to capture the overall structure of the gesture. By calculating the Euclidean distance between any pair of skeletal key points, a symmetric matrix can be obtained, and the lower triangular matrix without the diagonal part of the symmetric matrix is ​​used to represent the first joint distance feature. For example, assuming that the total number of frames is T, there are N joints, and at a certain frame t, the Cartesian coordinates of the nth joint are By gathering all joint coordinates, we can get the joint coordinate set Then the joint coordinate set C of the tth frame is t The calculation formula of the joint distance is:

[0083]

[0084] in, express and The Euclidean distance between t Represents the first joint distance feature of the t-th frame.

[0085] A symmetric matrix is ​​constructed by calculating the Euclidean distance between joints. This geometric feature is position-viewpoint invariant and is therefore not affected by skeleton rotation or movement. It only uses the lower triangular matrix without the diagonal part, that is, it does not include diagonal elements, which reduces feature redundancy and improves model speed.

[0086] In step S230 of some embodiments, the gesture is described from the visual global motion, and the first dual-scale motion feature formed combines the characteristics of fast motion and slow motion, avoiding misjudgment caused by different individual motion speeds. Then the first dual-scale motion feature includes a first slow motion feature and a first fast motion feature. By inputting the first slow motion feature and the first fast motion feature, the gesture recognition network with multi-feature fusion can pay attention to fast motion and slow motion at the same time. The first dual-scale motion feature can be generated by the following equation:

[0087]

[0088]

[0089] in, represents the slow motion of the tth frame; Indicates the fast motion of frame t.

[0090] Since most gesture movements are related to global trajectories, global motion features need to be considered. By obtaining the first dual-scale motion features and combining the two characteristics of fast motion and slow motion, global motion information of different scales and good generalization is obtained, avoiding misjudgment caused by different individual motion speeds.

[0091] In step S300 of some embodiments, in the EMG signal branch of the multimodal gesture recognition network, two transformation features, the original features of the EMG signal and the change features of the EMG signal, are formed to describe the gesture from the intrinsic regularity of the data and the rapid dynamic changeability, respectively. Combining the original features of the EMG signal and the change features of the EMG signal, the original features are used to provide a macroscopic perspective, and the change features are used to provide microscopic details, which is conducive to multi-scale analysis. Through these two features, different aspects of the gesture can be captured, and complementary information can be provided. The comprehensive use can improve the accuracy and reliability of gesture recognition, and thus a more comprehensive and robust gesture recognition system can be constructed.

[0092] In some embodiments, step S300 may include but is not limited to steps S310 to S320:

[0093] Step S310, obtaining original features of the first electromyographic signal according to the electromyographic signal;

[0094] Step S320: Acquire the first electromyographic signal change feature of the current frame according to the first electromyographic signal original feature of the current frame and the first electromyographic signal original feature of the next frame.

[0095] In step S310 of some embodiments, the original features of the electromyographic signal, i.e., the electromyographic signal in the original format, are obtained to describe the gesture from the inherent regularity of the data. The original features of the electromyographic signal include complete sequence data of eight channels, which is essential for capturing the overall pattern and long-term trend of the gesture.

[0096] In step S320 of some embodiments, the gesture is described from the perspective of rapid dynamic variability, and the electromyographic signal change feature emphasizes the rapid changes associated with muscle activity through a first-order difference operation. For example, muscle contraction and relaxation may cause significant fluctuations in the surface electromyographic signal, and the electromyographic signal change feature can better capture these changes. Exemplarily, by calculating the first electromyographic signal change feature using a first-order time difference, the muscle movement characteristics can be captured, further enhancing the generalization ability of the model. Then the first electromyographic signal change feature is defined as:

[0097] SVF t =O t+1 -O t ,t∈1,2,…,T-1

[0098] Among them, SVF t Represents the first electromyographic signal change feature of the current frame; t Represents the original feature of the first electromyographic signal of the current frame; t+1 Represents the original feature of the first electromyographic signal of the next frame; t represents the tth frame; T represents the total number of frames.

[0099] The dynamic changes of EMG signals are crucial for applications such as motion pattern recognition, as they can more accurately reflect muscle activity during hand movements. By acquiring the EMG signal change characteristics, the rapid dynamic changes of the signal can be captured; by combining the EMG signal change characteristics, the influence of overfitting in the modeling process due to the redundant information that may be contained in the original EMG signal characteristics can be reduced; through differential operations, low-frequency noise and baseline drift can be effectively suppressed, and signal components related to actual muscle activity can be highlighted. In addition, through the differential feature of the EMG signal change characteristics, it is easier to capture the timing information during the gesture execution process, such as the start, duration, and end of the action, thereby enhancing the sensitivity of the information in the time dimension.

[0100] In step S400 of some embodiments, the first joint distance feature, the first dual-scale motion feature, the first electromyographic signal original feature and the first electromyographic signal change feature are respectively embedded into the latent vector of each frame, and the embedded features corresponding to the first joint distance feature, the first dual-scale motion feature, the first electromyographic signal original feature and the first electromyographic signal change feature are obtained to obtain the second joint distance feature, the second dual-scale motion feature, the second electromyographic signal original feature and the second electromyographic signal change feature.

[0101] In some embodiments, Figure 2 As shown, the embedding layer can include 3 one-dimensional convolutional layers and a maximum pooling layer. Local feature extraction is performed through the one-dimensional convolutional layer. Each one-dimensional convolutional layer operates on the input data through a sliding window (convolution kernel) to capture local patterns and features. In the maximum pooling layer, the spatial dimension of the feature is reduced while retaining the most important information. It is achieved by selecting the maximum value within a certain window. This can reduce the number of parameters, reduce the risk of overfitting, and improve the translation invariance of the model to a certain extent.

[0102] In some embodiments, different gesture actions may show different key point / joint point correlations. Therefore, by using an additional embedding layer, the two transformation features of joint distance feature and dual-scale motion feature are embedded into the latent vector of each frame, so that the network can learn the correlation between joint points while reducing the influence of noise in the skeleton data. Exemplarily, the first joint distance feature is input into the embedding layer, and after passing through the one-dimensional convolution layer and the maximum pooling layer, the second joint distance feature can be obtained; the first slow motion feature in the first dual-scale motion feature is input into the embedding layer, and after passing through the one-dimensional convolution layer and the maximum pooling layer, the second slow motion feature can be obtained; the first fast motion feature in the first dual-scale motion feature is input into the embedding layer, and after passing through the one-dimensional convolution layer and the maximum pooling layer, the second fast motion feature can be obtained; according to the second slow motion feature and the second fast motion feature, the second dual-scale motion feature can be obtained.

[0103] In some embodiments, both the EMG signal change features and the EMG signal original features are embedded into the latent vector of each frame by using an additional embedding layer. By embedding these two types of EMG signal conversion features, the multimodal / multi-feature gesture recognition model can automatically learn the inter-channel correlation between the 8 channels of the EMG signal and reduce the noise interference in the EMG signal. Among them, the original information of the EMG signal, including the complete sequence data of the eight channels, is crucial to capturing the overall pattern and long-term trend of the gesture. Feeding the raw data directly into the embedding layer allows the embedding layer network to autonomously learn the most relevant features, which may include but are not limited to: the synergy pattern between different muscle groups; the overall distribution and variation of signal strength; frequency domain features, such as dominant frequency and harmonic structure; phase relationship and time correlation between multiple channels. This method retains the comprehensive information of the signal, enabling the model to discover subtle patterns that people may overlook. Exemplarily, the first electromyographic signal change feature is input into the embedding layer, and after passing through the one-dimensional convolution layer and the maximum pooling layer, the second electromyographic signal change feature can be obtained; the first electromyographic signal original feature is input into the embedding layer, and after passing through the one-dimensional convolution layer and the maximum pooling layer, the second electromyographic signal original feature can be obtained.

[0104] In steps S500 to S600 of some embodiments, since the four transformation features of the second joint distance feature, the second dual-scale motion feature, the second electromyographic signal original feature, and the second electromyographic signal change feature have different contributions to gesture recognition, it is necessary to further enhance the feature performance by explicitly modeling the relationship between the four features. A simple channel attention module can be used to assign weights to all channels in the total feature (i.e., the four transformation features included in the skeletal visual branch and the electromyographic signal branch), and then the updated target embedding feature is derived through weighted aggregation to obtain the posture classification result.

[0105] In some embodiments, step S500 may include but is not limited to steps S510 to S540:

[0106] Step S510, connecting the second joint distance feature, the second dual-scale motion feature, the second electromyographic signal original feature, and the second electromyographic signal change feature in the channel dimension to obtain a total feature;

[0107] Step S520, processing the total features through a global average pooling layer, a fully connected layer, and a sigmoid function to generate an initial channel attention map;

[0108] Step S530, in the channel dimension, adjusting the weight ratio of the initial channel attention map in each spatial dimension to obtain an intermediate channel attention map;

[0109] Step S540, adding the intermediate channel attention map to the total feature to obtain the target channel attention map.

[0110] In step S510 of some embodiments, since the four processed transformation features of the second joint distance feature, the second dual-scale motion feature, the second electromyographic signal original feature, and the second electromyographic signal change feature are represented as a one-dimensional signal in each frame, these features can be connected in the channel dimension to form a total feature F total The total number of channels of all features is denoted by C, which includes the four transformed features after being processed by the embedding layer.

[0111] In step S520 of some embodiments, Figure 2 As shown in Figure 1, the total feature is passed through the global pooling layer in the fusion module to obtain the spatial information of each channel in the total feature and generate a channel-level feature map of size 1×1×C. Subsequently, multiple fully connected layers learn channel information from the spatially reduced feature map. These features are passed through the sigmoid function to generate the initial channel attention map F′ total ∈R 1*1*C , whose value is in the range of (0,1), is expressed as:

[0112] F′ total =Sigmoid(FC(GAP(F total )))

[0113] Among them, F′ total represents the initial channel attention map; F total represents the total feature; FC represents the fully connected layer; GAP represents the global average pooling.

[0114] In steps S530 to S540 of some embodiments, Figure 2 As shown, the initial channel attention map F'total ∈R 1*1*C It will be multiplied by H*W*C, which is equivalent to adjusting the weight ratio of each H*W value in the C channel. After the 1*1*C initial channel attention map is multiplied by H*W*C, the weighted result can be obtained, that is, the intermediate channel attention map can be obtained, and the dimension of the intermediate channel attention map is H*W*C. Then the intermediate channel attention map is added to the total feature to obtain the target channel attention map. For example, in the channel dimension, the initial channel attention map F′ in each spatial dimension total Adjust the weight ratio to get the middle channel attention map F″ total ∈R H*W*C Next, the middle channel attention map F″ total With the total feature F total Add them together to get the target channel attention map F final , then we have the following expression:

[0115]

[0116] In the formula, F final represents the target channel attention map; F″ total represents the middle channel attention map; Stands for element-wise addition.

[0117] In some embodiments, step S600 may include but is not limited to steps S610 to S620:

[0118] Step S610, performing convolution and pooling operations on the target channel attention map through a temporal convolution block to obtain an intermediate feature map;

[0119] Step S620, passing the intermediate feature map through a fully connected layer to obtain the gesture recognition result.

[0120] In some embodiments, in steps S610 to S620, updated target embedding features are derived by weighted aggregation to obtain gesture classification results. That is, rich time information is extracted through temporal convolution blocks (multiple one-dimensional convolutions and global average pooling), and the final gesture category prediction is obtained through the full connection layer. Then the expression is:

[0121] F result =FC(GAP(Conv1d(F final )))

[0122] Among them, F result Represents the target embedding feature; Conv1d represents one-dimensional convolution.

[0123] In the existing research on gesture recognition technology, most of them are single-modal recognition, and the use of single-modal recognition always has limitations. For example, only using image information for recognition, this vision-based method is low-cost and easy to deploy, but sensitive to complex backgrounds (such as occlusion problems); only using sensor glove data for recognition, the data accuracy is high but the equipment cost is high and data collection is troublesome; only using surface electromyography (sEMG) for gesture recognition can capture the gesture time information or spatial information of subtle muscle activities in the wrist with high accuracy and sensitivity, but this method is easily affected by signal noise. In view of this, the embodiment of the present invention is based on different modalities to represent different physical information, and uses a combination of multiple modalities to make the model more universal and generalized. For example, from a visual perspective, the simple action of raising your hand can be identified as a selfie or badminton posture. In this case, it is difficult to distinguish the two actions visually, but they involve different muscle strengths. Therefore, the intensity information of the electromyographic signal can be used as a key factor in distinguishing these two seemingly similar actions. For another example, when the hand position in the image is blocked by other objects due to environmental factors, the electromyographic signal will not be affected by the blockage. Therefore, the combination of electrical signals and visual information will also improve the accuracy of gesture recognition and make it more generalizable.

[0124] In addition, some studies have constructed multimodal data sets and multimodal fusion gesture recognition methods for gesture recognition, but there is still room for improvement in the "high running speed + high precision" requirements of gesture recognition multimodal models. At the same time, multimodal gesture recognition models are rarely implemented in actual robot grasping and teleoperation. In view of this, the embodiment of the present invention designs a multimodal (visual information + electromyographic signal) gesture recognition model with fast inference speed and high recognition accuracy, and successfully deploys this model to the actual robot grasping and operating system to effectively perform robot teleoperation tasks.

[0125] like Figure 3As shown, the embodiment of the present invention successfully builds a comprehensive system involving human-computer interaction and robot remote operation in the real world based on the proposed multimodal gesture recognition network, so that the robot can imitate humans to perform complex grasping tasks. The robot remote operation system of the embodiment of the present invention is mainly composed of MYO armband, RealSense D435i camera, RealMan humanoid mechanical arm, Inspire humanoid mechanical dexterous hand and multimodal gesture recognition network. On the one hand, the position and direction must be controlled, so the three-dimensional angle information of the IMU (inertial measurement unit) is combined with the three-dimensional posture information of the visual system to achieve 6 degrees of freedom control. Specifically, since the IMU is prone to drift, resulting in inaccurate position, the present invention uses the RealSense camera to obtain RGB and depth video. RGB is processed by pose estimation to obtain three-dimensional coordinates, while the depth corrects the less accurate z dimension estimate. The IMU of the MYO armband is used to obtain three-dimensional rotation coordinates. Finally, the combined translation and rotation data are smoothed by a Kalman filter. The filtered data is then transmitted to the RealMan humanoid mechanical arm. After receiving this data, the robot arm calculates the data of each joint axis according to inverse kinematics to perform the movement. At this stage, the operator's hand movement is minimal, allowing the robot arm to operate in motion mode.

[0126] In addition, the 2D pose estimation (only x,y coordinates) of the visual image and the electromyographic signal are used as input to the recognition network to determine the gesture category, which is then transmitted to the Inspire humanoid robot hand to perform the corresponding gesture. A threshold parameter is also set in this system to calculate the rate of change of the hand joint angle, which indicates the degree of hand movement. Once this threshold is exceeded, the robot teleoperation system switches to the dexterous hand movement mode and transmits the gesture to the dexterous hand through the multimodal gesture recognition network, enabling it to perform complex grasping tasks.

[0127] The embodiment of the present invention further provides a gesture recognition device by fusion of electromyography and vision multiple features, which can implement the above-mentioned gesture recognition method by fusion of electromyography and vision multiple features. The device includes:

[0128] The first module is used to collect visual signals and electromyographic signals;

[0129] A second module is used to obtain a first joint distance feature and a first dual-scale motion feature according to the visual signal;

[0130] The third module is used to obtain the original characteristics of the first electromyographic signal and the change characteristics of the first electromyographic signal according to the electromyographic signal;

[0131] A fourth module is used to input the first joint distance feature, the first dual-scale motion feature, the first electromyographic signal original feature, and the first electromyographic signal change feature into an embedding layer respectively to obtain a second joint distance feature, a second dual-scale motion feature, a second electromyographic signal original feature, and a second electromyographic signal change feature;

[0132] A fifth module is used to input the second joint distance feature, the second dual-scale motion feature, the second electromyographic signal original feature and the second electromyographic signal change feature into a fusion module to obtain a target channel attention map;

[0133] The sixth module is used to obtain a gesture recognition result according to the target channel attention map.

[0134] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0135] The embodiment of the present invention further provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the above-mentioned gesture recognition method of electromyography-vision multi-feature fusion is implemented. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0136] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0137] refer to Figure 4 , Figure 4 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0138] The processor 701 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0139] The memory 702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 702, and the processor 701 calls and executes a gesture recognition method of electromyography-vision multi-feature fusion according to an embodiment of the present invention;

[0140] Input / output interface 703, used to implement information input and output;

[0141] Communication interface 704, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0142] A bus 705 that transmits information between the various components of the device (e.g., the processor 701, the memory 702, the input / output interface 703, and the communication interface 704);

[0143] The processor 701 , the memory 702 , the input / output interface 703 and the communication interface 704 are connected to each other in communication within the device via a bus 705 .

[0144] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned gesture recognition method of electromyography-vision multi-feature fusion.

[0145] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0146] The embodiment of the present invention also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned gesture recognition method of electromyography-vision multi-feature fusion.

[0147] In summary, the method and device for gesture recognition based on myoelectric vision multi-feature fusion according to the embodiment of the present invention have the following advantages:

[0148] 1. The embodiment of the present invention proposes a multimodal gesture recognition network that can convert two modal information (vision + surface electromyography sEMG) into four transformation features, including: joint distance feature, dual-scale motion feature, electromyography original feature, and electromyography change feature. The multimodal gesture recognition network then uses these four features for processing and performs feature fusion based on channel attention, thereby enhancing the model's ability to understand gestures and improving the accuracy and speed of gesture recognition.

[0149] 2. The embodiment of the present invention constructs a robot remote operation system, which uses the proposed multimodal gesture recognition model to drive the robot to perform grasping operations. The inertial measurement unit (IMU) in the MYO armband and the information of visual estimation based on the depth camera are jointly processed to obtain stable six-dimensional data of the end of the robotic arm (x, y, z, rx, ry, rz, that is, three-dimensional coordinates + three rotation angles), which can control the movement of the end of the robotic arm. The eight-channel electromyographic signals and visual data in the MYO armband are simultaneously fed into the multimodal gesture recognition model to obtain dynamic gesture recognition results, which are then transmitted to the mechanical dexterous hand to control the dexterous hand to make corresponding actions, and can perform complex grasping tasks involving multiple gesture combinations, and interact naturally with users in real life.

[0150] 3. Compared with the common gesture recognition method based on single mode, the method proposed in the embodiment of the present invention utilizes multi-modal information, uses electromyographic data as information supplement to visual data, and can achieve gesture recognition with higher accuracy. Compared with many similar multi-modal gesture recognition methods, the method proposed in the embodiment of the present invention is designed to achieve high performance of the model, achieve fast gesture recognition reasoning while ensuring high accuracy, and is easy to deploy to a variety of actual robot systems.

[0151] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0152] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0153] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0154] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0155] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0156] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0157] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0158] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0159] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A gesture recognition method based on electromyography and visual multi-feature fusion, characterized in that: The following steps are involved: Collect visual signals and electromyographic signals; Acquire a first joint distance feature and a first dual-scale motion feature according to the visual signal; According to the electromyographic signal, obtaining the original feature of the first electromyographic signal and the change feature of the first electromyographic signal; Input the first joint distance feature, the first dual-scale motion feature, the first electromyographic signal original feature, and the first electromyographic signal change feature into the embedding layer respectively to obtain a second joint distance feature, a second dual-scale motion feature, a second electromyographic signal original feature, and a second electromyographic signal change feature; Input the second joint distance feature, the second dual-scale motion feature, the second electromyographic signal original feature, and the second electromyographic signal change feature into a fusion module to obtain a target channel attention map; According to the target channel attention map, a gesture recognition result is obtained.

2. The method for gesture recognition based on electromyography and multi-feature fusion according to claim 1, characterized in that: The collecting of visual signals and electromyographic signals comprises the following steps: Capturing RGB video through a first device; Performing posture estimation on the RGB video to obtain the visual signal; The electromyographic signal is collected by a second device.

3. The method for gesture recognition based on electromyography and visual multi-feature fusion according to claim 1, characterized in that: The step of acquiring a first joint distance feature and a first dual-scale motion feature according to the visual signal comprises the following steps: Obtaining the Euclidean distance between any pair of skeleton key points in the visual signal and constructing a symmetric matrix; Obtaining the first joint distance feature according to the lower triangular matrix of the symmetric matrix; Acquire the first dual-scale motion feature according to the frame number and the joint coordinate set corresponding to the frame number; The first dual-scale motion feature includes a first slow motion feature and a first fast motion feature.

4. The method for gesture recognition based on myoelectric visual multi-feature fusion according to claim 1, characterized in that: The step of obtaining the original feature of the first electromyographic signal and the change feature of the first electromyographic signal according to the electromyographic signal comprises the following steps: According to the electromyographic signal, obtaining the original feature of the first electromyographic signal; According to the original features of the first electromyographic signal of the current frame and the original features of the first electromyographic signal of the next frame, the change features of the first electromyographic signal of the current frame are obtained.

5. The method for gesture recognition based on myoelectric visual multi-feature fusion according to claim 4 is characterized in that: The formula used for obtaining the first electromyographic signal change feature of the current frame according to the first electromyographic signal original feature of the current frame and the first electromyographic signal original feature of the next frame includes: SVF t =O t+1 -O t ,t∈1,2,…,T-1 Among them, SVF t Represents the first electromyographic signal change feature of the current frame; t Represents the original feature of the first electromyographic signal of the current frame; t+1 Represents the original feature of the first electromyographic signal of the next frame; t represents the tth frame; T represents the total number of frames.

6. The method for gesture recognition based on electromyography and visual multi-feature fusion according to claim 1, characterized in that: The first joint distance feature, the first dual-scale motion feature, the first electromyographic signal original feature, and the first electromyographic signal change feature are respectively input into the embedding layer to obtain a second joint distance feature, a second dual-scale motion feature, a second electromyographic signal original feature, and a second electromyographic signal change feature, including the following steps: The first joint distance feature, the first dual-scale motion feature, the first electromyographic signal original feature and the first electromyographic signal change feature are respectively embedded into the potential vector of each frame, and the embedded features corresponding to the first joint distance feature, the first dual-scale motion feature, the first electromyographic signal original feature and the first electromyographic signal change feature are obtained respectively, and the second joint distance feature, the second dual-scale motion feature, the second electromyographic signal original feature and the second electromyographic signal change feature are obtained.

7. The method for gesture recognition based on electromyography and visual multi-feature fusion according to claim 1, characterized in that: The step of inputting the second joint distance feature, the second dual-scale motion feature, the second electromyographic signal original feature, and the second electromyographic signal change feature into a fusion module to obtain a target channel attention map comprises the following steps: Connecting the second joint distance feature, the second dual-scale motion feature, the second electromyographic signal original feature, and the second electromyographic signal change feature in the channel dimension to obtain a total feature; The total features are processed through a global average pooling layer, a fully connected layer, and a sigmoid function to generate an initial channel attention map; In the channel dimension, the weight ratio of the initial channel attention map in each spatial dimension is adjusted to obtain an intermediate channel attention map; The intermediate channel attention map is added to the total feature to obtain the target channel attention map.

8. The method for gesture recognition based on electromyography and visual multi-feature fusion according to claim 1, characterized in that: Obtaining a gesture recognition result according to the target channel attention map includes the following steps: The target channel attention map is convolved and pooled through a temporal convolution block to obtain an intermediate feature map; The intermediate feature map is passed through a fully connected layer to obtain the gesture recognition result.

9. A gesture recognition device based on myoelectric vision multi-feature fusion, characterized in that: include: The first module is used to collect visual signals and electromyographic signals; A second module is used to obtain a first joint distance feature and a first dual-scale motion feature according to the visual signal; The third module is used to obtain the original characteristics of the first electromyographic signal and the change characteristics of the first electromyographic signal according to the electromyographic signal; A fourth module is used to input the first joint distance feature, the first dual-scale motion feature, the first electromyographic signal original feature, and the first electromyographic signal change feature into an embedding layer respectively to obtain a second joint distance feature, a second dual-scale motion feature, a second electromyographic signal original feature, and a second electromyographic signal change feature; A fifth module is used to input the second joint distance feature, the second dual-scale motion feature, the second electromyographic signal original feature and the second electromyographic signal change feature into a fusion module to obtain a target channel attention map; The sixth module is used to obtain a gesture recognition result according to the target channel attention map.

10. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 8.

Images