A multimodal gesture recognition method and system based on electromyography and vision fusion

By combining electromyography and visual data, the joint joint distance and motion characteristics are calculated, the problems of gesture recognition in environmental impact and individual dependence are solved, and high-precision and high-real-time gesture recognition are achieved.

CN118520419BActive Publication Date: 2024-11-26SUN YAT SEN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410978487.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-11-26
Estimated Expiration
2044-07-22

AI Technical Summary

Technical Problem

The existing gesture recognition technology has defects in environmental impact, individual dependence and noise interference, resulting in low recognition accuracy.

Method used

Using multimodal gesture recognition method based on electromyography and vision, the joint joint distance characteristics, slow motion characteristics and fast motion characteristics are calculated by obtaining the bone point information and electromyography signals of the hand image, and the joint joint distance characteristics are calculating, and the joint filtering and feature fusion are carried out through threshold filtering and feature fusion. Processing improves identification accuracy.

Benefits of technology

It realizes higher accuracy and more generalized gesture recognition, enhances the model's understanding of gestures, improves the accuracy of recognition, and has high real-time performance, suitable for real-time interaction of human-computer interaction and robotic arms.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118520419B_ABST
    Figure CN118520419B_ABST
Patent Text Reader

Abstract

The present application discloses a multimodal gesture recognition method and system based on electromyography and visual fusion, including: obtaining the skeleton point information and the first surface electromyography signal of the hand image; calculating the joint distance feature, slow motion feature and fast motion feature according to the skeleton point information; performing denoising processing on the first surface electromyography signal by threshold filtering to obtain the second surface electromyography signal; extracting the joint distance feature, slow motion feature, fast motion feature and the feature map of the second surface electromyography signal by embedding operation, and correspondingly obtaining the joint distance feature map, slow motion feature map, fast motion feature map and surface electromyography signal feature map; performing feature fusion processing on these feature maps to obtain a fusion feature map; obtaining the gesture recognition result according to the fusion feature map. The present application can realize higher precision and more general gesture recognition by fusing electromyography signal and visual image data, and can be widely used in the field of computer technology.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a multimodal gesture recognition method and system based on electromyography and vision fusion. Background Art

[0002] Gestures are a form of communication widely used by people in social communication. People use gestures to express various information. Gesture recognition refers to the technology of capturing the posture and shape information of human hands through instruments such as cameras, recognizing them through computers, and converting them into various operation instructions. According to the movement characteristics of gestures, gestures can be divided into static gestures and dynamic gestures. Static gestures refer to gestures in which the shape and position of the hand remain unchanged and the information is conveyed through the outline of the hand. This type of gesture is only related to the characteristics of a single frame image, so only the single frame image needs to be recognized and processed during processing; dynamic gestures refer to changes in the overall position and joint position of the hand, which contain richer information. This type of gesture requires the acquisition of video sequences for inter-frame processing.

[0003] Among the related technologies, gesture recognition technology is mainly divided into two categories: gesture recognition technology based on visual images, which mainly relies on cameras for gesture recognition and tracking, and technology based on wearable devices, which obtains hand posture information for recognition through various devices worn by users on their hands or arms, such as gloves and bracelets.

[0004] However, single image information still has its defects, such as being easily affected by the shooting environment, poor lighting environment, partial occlusion of the hand, complex shooting background and other factors that will seriously affect the recognition accuracy. The recognition method of obtaining the surface electromyographic signal of the hand also has some inherent defects, such as individual dependence, the strength and form of each individual's electromyographic signal are different and non-stationary, there are many types of noise, and the signal fluctuates greatly. In addition, in practical applications, electromyographic signals will introduce interference due to muscle fatigue and skin sweating, which will also affect the recognition accuracy. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a multimodal gesture recognition method and system based on electromyography and vision fusion, aiming to achieve gesture recognition with high accuracy and good versatility.

[0006] To achieve the above object, an embodiment of the present application proposes a multimodal gesture recognition method based on electromyography and vision fusion, the method comprising:

[0007] Obtaining skeleton point information and first surface electromyography signal of the hand image;

[0008] Calculate joint distance features, slow motion features and fast motion features according to the skeleton point information;

[0009] Performing denoising processing on the first surface electromyography signal by threshold filtering to obtain a second surface electromyography signal;

[0010] Extracting the joint distance feature, the slow motion feature, the fast motion feature and the feature map of the second surface electromyography signal through an embedding operation, and obtaining a joint distance feature map, a slow motion feature map, a fast motion feature map and a surface electromyography signal feature map accordingly;

[0011] Performing feature fusion processing on the joint distance feature map, the slow motion feature map, the fast motion feature map and the surface electromyography signal feature map to obtain a fused feature map;

[0012] A gesture recognition result is obtained according to the fused feature map.

[0013] In some embodiments, the step of acquiring the skeleton point information and the first surface electromyography signal of the hand image comprises the following steps:

[0014] Acquire a hand video; wherein the hand video includes several frames of hand images;

[0015] Using a hand gesture and key point recognition algorithm to identify skeleton point information based on the hand image;

[0016] A first surface electromyography signal of the hand is collected by a first device.

[0017] In some embodiments, the step of calculating the joint distance feature, the slow motion feature and the fast motion feature according to the skeleton point information comprises the following steps:

[0018] Determine the Cartesian coordinates of each joint according to the skeleton point information;

[0019] Determine a joint distance feature according to the number of frames, the number of joint points and the Cartesian coordinates;

[0020] Determine a slow motion feature according to current skeleton point information of a current frame and first skeleton point information of a first adjacent frame; wherein the first adjacent frame is directly adjacent to the current frame;

[0021] The fast motion feature is determined according to the current skeleton point information of the current frame and the second skeleton point information of the second adjacent frame; wherein the second adjacent frame is one frame away from the current frame.

[0022] In some embodiments, performing denoising processing on the first surface electromyography signal by threshold filtering to obtain a second surface electromyography signal comprises the following steps:

[0023] Performing discrete wavelet transform processing on the first surface electromyography signal to obtain a first processing result;

[0024] Performing threshold processing on the first processing result to obtain a second processing result;

[0025] The second processing result is subjected to inverse discrete wavelet transform processing to obtain a second surface electromyography signal.

[0026] In some embodiments, obtaining a gesture recognition result according to the fused feature map comprises the following steps:

[0027] Using a one-dimensional convolution block to extract features from the fused feature map to obtain a third processing result;

[0028] Performing a first activation operation and a batch normalization operation on the third processing result to obtain a fourth processing result;

[0029] Performing global average pooling processing on the fourth processing result to obtain a fifth processing result;

[0030] Performing full connection processing on the fifth processing result to obtain a sixth processing result;

[0031] A second activation operation is performed on the sixth processing result to obtain a gesture recognition result.

[0032] In some embodiments, the embedding operation uses a convolution block and / or a pooling layer, and the expression of the convolution block is:

[0033] ;

[0034] in, Represents a one-dimensional convolution kernel with a convolution kernel size of 1 and a number of channels. ; Indicates the filter value.

[0035] In some embodiments, the expression of the feature fusion processing is:

[0036] ;

[0037] in, represents the k-th frame fusion feature map, represents a fusion operation; Indicates that the kth frame is embedded by the operation The extracted joint distance features; Indicates that the kth frame is embedded by the operation The extracted slow motion features; Indicates that the kth frame is embedded by the operation The extracted fast motion features; Indicates that the kth frame is embedded by the operation Extracted surface electromyography signal features.

[0038] To achieve the above purpose, another aspect of the embodiment of the present application proposes a multimodal gesture recognition system based on electromyography and vision fusion, the system comprising:

[0039] The first module is used to obtain the skeleton point information and the first surface electromyography signal of the hand image;

[0040] The second module is used to calculate the joint distance feature, the slow motion feature and the fast motion feature according to the skeleton point information;

[0041] A third module is used to perform denoising on the first surface electromyography signal by threshold filtering to obtain a second surface electromyography signal;

[0042] A fourth module is used to extract the joint distance feature, the slow motion feature, the fast motion feature and the feature map of the second surface electromyography signal through an embedding operation, and obtain a joint distance feature map, a slow motion feature map, a fast motion feature map and a surface electromyography signal feature map accordingly;

[0043] A fifth module is used to perform feature fusion processing on the joint distance feature map, the slow motion feature map, the fast motion feature map and the surface electromyography signal feature map to obtain a fusion feature map;

[0044] The sixth module is used to obtain a gesture recognition result according to the fused feature map.

[0045] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0046] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0047] The embodiments of the present application include at least the following beneficial effects: The present application provides a multimodal gesture recognition method and system based on electromyography and visual fusion, the scheme obtains the skeleton point information and the first surface electromyography signal of the hand image; calculates the joint distance feature, slow motion feature and fast motion feature according to the skeleton point information; performs denoising processing on the first surface electromyography signal through threshold filtering to obtain the second surface electromyography signal; extracts the joint distance feature, slow motion feature, fast motion feature and the feature map of the second surface electromyography signal through embedding operation, and obtains the joint distance feature map, slow motion feature map, fast motion feature map and surface electromyography signal feature map accordingly; performs feature fusion processing on the joint distance feature map, slow motion feature map, fast motion feature map and surface electromyography signal feature map to obtain a fusion feature map; according to the fusion feature map, obtains the overall steps of the gesture recognition result, fuses the electromyography data with the visual data, and the two data information complement each other. Combined with the joint distance feature, slow motion feature and fast motion feature, it can achieve higher accuracy and more general gesture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0049] Figure 1 is a flow chart of a multimodal gesture recognition method based on myoelectricity and vision fusion provided in an embodiment of the present application;

[0050] Figure 2 is a wavelet threshold denoising flow chart provided in an embodiment of the present application;

[0051] Figure 3 It is a multimodal gesture recognition network structure provided by an embodiment of the present application;

[0052] Figure 4 is an application architecture block diagram of multimodal gesture recognition provided by an embodiment of the present application;

[0053] Figure 5 It is a module schematic diagram of a multimodal gesture recognition system based on myoelectricity and vision fusion provided in an embodiment of the present application;

[0054] Figure 6 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0056] Although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100", "second / S200", etc. in the specification, claims and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0057] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0058] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0059] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0061] In the related technologies, gesture recognition technology is mainly divided into two categories: gesture recognition technology based on visual images, which mainly relies on cameras for gesture recognition and tracking, and technology based on wearable devices, which obtains hand posture information for recognition through various devices worn by users on their hands or arms, such as gloves and bracelets. However, single image information still has its defects, such as being easily affected by the shooting environment, poor lighting environment, partial occlusion of the hands, complex shooting background and other factors that will seriously affect the recognition accuracy. The recognition method of obtaining the surface electromyographic signal of the hand also has some inherent defects, such as individual dependence, differences and non-stationarity in the strength and form of the electromyographic signal of each individual, many types of noise, and large signal fluctuations. In addition, in practical applications, electromyographic signals will introduce interference due to muscle fatigue and skin sweating, which will also affect the recognition accuracy.

[0062] In view of this, a multimodal gesture recognition method and system based on electromyography and vision fusion is provided in an embodiment of the present application. The scheme utilizes multimodal information to fuse electromyography data with visual data, so that the data information complements each other, and can achieve higher precision and more general gesture recognition. In addition, the joint distance (geometric features), dual-scale motion features (global features) and electromyography signal features are fused to enhance the model's ability to understand gestures and further improve the accuracy of gesture recognition. In addition, the method of the embodiment of the present application also has high real-time performance and can be used as a recognition method for human-computer interaction. For example, the combination of vision, electromyography and gyroscope data in IMU (inertial measurement unit) can be used to achieve real-time interaction between manipulators and robotic arms.

[0063] A multimodal gesture recognition method based on electromyography and visual fusion provided in an embodiment of the present application relates to the field of computer technology. A multimodal gesture recognition method based on electromyography and visual fusion provided in an embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The server can also be a node server in a blockchain network; the software can be an application that implements a multimodal gesture recognition method based on electromyography and visual fusion, etc., but is not limited to the above forms.

[0064] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0065] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0066] Figure 1 is an optional flow chart of a multimodal gesture recognition method based on myoelectricity and vision fusion provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S100 to S600.

[0067] Step S100, obtaining skeleton point information and a first surface electromyography signal of a hand image.

[0068] Step S200, calculating joint distance features, slow motion features and fast motion features according to the skeleton point information.

[0069] Step S300: denoising the first surface electromyography signal by threshold filtering to obtain a second surface electromyography signal.

[0070] Step S400, extracting the joint distance feature, the slow motion feature, the fast motion feature and the feature map of the second surface electromyography signal through an embedding operation, and correspondingly obtaining a joint distance feature map, a slow motion feature map, a fast motion feature map and a surface electromyography signal feature map.

[0071] Step S500, performing feature fusion processing on the joint distance feature map, the slow motion feature map, the fast motion feature map and the surface electromyography signal feature map to obtain a fused feature map.

[0072] Step S600, obtaining a gesture recognition result according to the fused feature map.

[0073] Steps S100 to S600 shown in the embodiment of the present application, by fusing electromyographic data with visual data, the two types of data information complement each other, and combined with joint distance features, slow motion features and fast motion features, can achieve higher-precision and more versatile gesture recognition.

[0074] In some embodiments, step S100 may include but is not limited to steps S110 to S130.

[0075] Step S110, obtaining a hand video; wherein the hand video includes several frames of hand images.

[0076] Step S120: using a hand gesture and key point recognition algorithm to recognize skeleton point information according to the hand image.

[0077] Step S130: collecting a first surface electromyography signal of the hand through a first device.

[0078] In some embodiments, step S200 may include but is not limited to steps S210 to S240.

[0079] Step S210, determining the Cartesian coordinates of each joint according to the skeleton point information.

[0080] Step S220, determining the joint distance feature of the joint according to the number of frames, the number of joint points and the Cartesian coordinates.

[0081] Step S230, determining the slow motion feature according to the current skeleton point information of the current frame and the first skeleton point information of the first adjacent frame; wherein the first adjacent frame is directly adjacent to the current frame.

[0082] Step S240, determining a fast motion feature according to the current skeleton point information of the current frame and the second skeleton point information of the second adjacent frame; wherein the second adjacent frame is one frame away from the current frame.

[0083] In some embodiments, step S300 may include but is not limited to steps S310 to S330.

[0084] Step S310: performing discrete wavelet transform processing on the first surface electromyography signal to obtain a first processing result.

[0085] Step S320: performing threshold processing on the first processing result to obtain a second processing result.

[0086] Step S330: performing inverse discrete wavelet transform processing on the second processing result to obtain a second surface electromyography signal.

[0087] In step S400 of some embodiments, the embedding operation uses a convolution block and / or a pooling layer, and the expression of the convolution block is:

[0088] ;

[0089] in, Represents a one-dimensional convolution kernel with a convolution kernel size of 1 and a number of channels. ; Indicates the filter value.

[0090] In step S500 of some embodiments, the expression of the feature fusion processing is:

[0091] ;

[0092] in, represents the k-th frame fusion feature map, represents a fusion operation; Indicates that the kth frame is embedded by the operation The extracted joint distance features; Indicates that the kth frame is embedded by the operation The extracted slow motion features; Indicates that the kth frame is embedded by the operation The extracted fast motion features; Indicates that the kth frame is embedded by the operation Extracted surface electromyography signal features.

[0093] In some embodiments, step S600 may include but is not limited to steps S410 to S450.

[0094] Step S610: Use a one-dimensional convolution block to extract features from the fused feature map to obtain a third processing result.

[0095] Step S620: Perform a first activation operation and a batch normalization operation on the third processing result to obtain a fourth processing result.

[0096] Step S630, performing global average pooling processing on the fourth processing result to obtain a fifth processing result.

[0097] Step S640: Perform full connection processing on the fifth processing result to obtain a sixth processing result.

[0098] Step S650: performing a second activation operation on the sixth processing result to obtain a gesture recognition result.

[0099] Below, in conjunction with a specific example of a gesture recognition scenario application, the solution of the embodiment of the present application is described in detail and explained:

[0100] In an embodiment of the present application, a multimodal gesture recognition method based on electromyography and vision fusion is provided. The method can be applied to perform gesture recognition in real time and accurately. The method may include the following six steps.

[0101] (1) Obtain the skeleton point information and the first surface electromyography signal of the hand image.

[0102] The RGB image of the hand is collected by the camera, and the collected RGB image is converted into skeleton point information. In some embodiments, mediapipe hands can be used for skeleton point collection. Mediapipe hands is a real-time hand posture and key point recognition algorithm that can use a single RGB camera to recognize the skeleton points of the human hand in real time.

[0103] The electromyographic signal can be collected by a portable collection device, which is called the first surface electromyographic signal. The portable collection device can be an electromyographic bracelet, etc.

[0104] (2) Calculate joint distance features, slow motion features, and fast motion features based on the skeleton point information.

[0105] After obtaining the skeleton point information, the skeleton point information is calculated as geometric features and global features: the geometric features are specifically manifested as joint distance features. The joint distance is the Euclidean distance between any pair of joints. In order to reduce redundancy, the embodiment of the present application only uses the lower triangular matrix without the diagonal part as the input of the joint distance. The data size is less than half of the original, thereby improving the recognition speed. Furthermore, when the total number of frames is K, there are N joints in total, and at a certain frame k, the Cartesian coordinates of joint i are , put all joint coordinates together to get the set ,but The calculation formula of the joint distance is:

[0106] ;

[0107] in, express and In the embodiment of the present application, the joint distance is flattened into a one-dimensional vector before being input into the network, and the vector length is .

[0108] Global features can also be called dual-scale global motion features, which can be divided into fast motion features (fastmotion) and slow motion features (slow motion). Through such input, the network can pay attention to both fast and slow motion. The dual-scale motion features can be generated by the following equation:

[0109] ;

[0110] .

[0111] in, represents the slow motion of the kth frame; represents the fast motion of the kth frame; Represents the skeleton point information of the kth frame.

[0112] For the skeleton point information, the joint distance is the distance matrix between the skeleton points, which reflects the relative displacement between the skeleton points, that is, the shape change of the hand. The dual-scale global motion feature can simultaneously identify the slow and fast movements of the hand. When the hand shape remains unchanged but the position changes, the dual-scale global motion feature plays a major role; when the hand shape changes but the position remains unchanged, the joint distance plays a major role. Combining the joint distance and the dual-scale global motion feature allows the network to simultaneously notice the changes in the shape and position of the hand.

[0113] (3) The first surface electromyography signal is denoised by threshold filtering to obtain a second surface electromyography signal.

[0114] The electromyographic signal collected by the electromyographic bracelet has noise at various frequencies. The embodiment of the present application adopts a wavelet threshold denoising method to decompose the signal into low-frequency components and high-frequency detail components, and perform threshold filtering on each component. The flowchart of wavelet threshold denoising can be referred to Figure 2 shown.

[0115] The basis of wavelet transform (WT) is generally a finite-length wavelet basis that decays, its energy is finite, and the value of the wavelet basis integral is zero. Wavelet transform can better reflect the change of frequency over time compared to Fourier transform. The embodiment of the present application obtains the time-frequency spectrum of the electromyographic signal through wavelet transform. Furthermore, the transformation formula of wavelet transform WT is:

[0116] ,

[0117] Among them, ψ represents the basis function of wavelet transform; a represents the scale, corresponding to frequency; b represents the translation, corresponding to time; t represents time. Common wavelet bases include Haar wavelet base, dbN wavelet base, mexh wavelet base, etc. The principle of wavelet transform is actually the conjugate convolution between the signal and the wavelet base.

[0118] After wavelet transform, the signal can be denoised by threshold processing.

[0119] (4) The joint distance feature, slow motion feature, fast motion feature and the feature map of the second surface electromyography signal are extracted through the embedding operation, and the joint distance feature map, slow motion feature map, fast motion feature map and surface electromyography signal feature map are obtained accordingly.

[0120] In the embodiment of the present application, a convolutional neural network (CNN) is selected as the basic architecture of the network. Figure 3 The multimodal gesture recognition network constructed in the embodiment of the present application is shown.

[0121] First, through the embedding operation, the joint distance features, dual-scale global motion features and feature maps of surface electromyography signals are extracted respectively. The expression of the convolution block in the embedding operation ε is:

[0122] ;

[0123] in, Represents a one-dimensional convolution kernel with a convolution kernel size of 1 and a number of channels. ,in, The value can be adjusted. Small, good network real-time performance, poor accuracy, The larger the network real-time performance, the better the accuracy. In the embodiment of the present application, in order to balance the real-time performance and accuracy, the general .

[0124] Through the embedding operation, the network can automatically learn the correlation between each joint and each surface EMG signal, and also prevent the network from ignoring the intrinsic coupling relationship of each feature.

[0125] (5) The joint distance feature map, slow motion feature map, fast motion feature map and surface electromyography signal feature map are subjected to feature fusion processing to obtain a fused feature map.

[0126] Since the frame rate of the fast motion feature is half of that of other features, the embodiment of the present application also uses maximum pooling to make the data length consistent. Then, these three features are fused through a concatenation operation, and the fused expression is:

[0127] ;

[0128] in, represents the k-th frame fusion feature map, represents a fusion operation; Indicates that the kth frame is embedded by the operation The extracted joint distance features; Indicates that the kth frame is embedded by the operation The extracted slow motion features; Indicates that the kth frame is embedded by the operation The extracted fast motion features; Indicates that the kth frame is embedded by the operation Extracted surface electromyography signal features.

[0129] (6) Based on the fused feature map, the gesture recognition result is obtained.

[0130] After fusion, a one-dimensional convolution block is used to perform further feature extraction on the fused feature map. The convolution block is defined as:

[0131] ;

[0132] Among them, / 2 represents the maximum pooling operation.

[0133] After each convolution operation, the activation operation and batch normalization operation are performed. The activation function can use the leaky relu function. Then, through the global average pooling layer (GAP) and the fully connected layer (FC), the softmax function is used to obtain the final gesture recognition result.

[0134] In some embodiments, the interaction mode of the system constructed based on the method of the embodiment of the present application can refer to Figure 4As shown in the figure, the working process is as follows: First, the camera collects the hand image information, and the skeleton point information is marked by the mediapipe hands, and the skeleton point information is processed into geometric features and global features. The surface electromyography signal of the forearm is collected by the electromyographic bracelet, and wavelet threshold denoising is performed. The information of the two modes is input into the multimodal recognition network for recognition, and the gesture recognition result is input into the robot to realize the robot to follow the human gesture. At the same time, the IMU sensor in the electromyographic bracelet is used to obtain the gyroscope information, which is converted into the pitch angle and yaw angle information of the arm. Through the joint motion control method, the robot arm is controlled by determining the rotation angle of each joint to realize the motion following of the robot arm.

[0135] In summary, the embodiments of the present application at least include the following beneficial effects:

[0136] The method of the embodiment of the present application utilizes multimodal information, and fuses electromyographic data with visual data, so that the data information complements each other, and can achieve higher precision and more general gesture recognition. In addition, the joint distance (geometric features), dual-scale motion features (global features) and electromyographic signal features are integrated to enhance the model's ability to understand gestures and further improve the accuracy of gesture recognition. The method of the embodiment of the present application also has high real-time performance and can be used as a recognition method for human-computer interaction. For example, vision, electromyographic signals and gyroscope data in the IMU (inertial measurement unit) are combined to achieve real-time interaction between manipulators and robotic arms.

[0137] See also Figure 5 The embodiment of the present application also provides a multimodal gesture recognition system based on electromyography and vision fusion, which can implement the above-mentioned multimodal gesture recognition method based on electromyography and vision fusion, and the system includes:

[0138] The first module 101 is used to obtain the skeleton point information and the first surface electromyography signal of the hand image;

[0139] The second module 102 is used to calculate the joint distance feature, the slow motion feature and the fast motion feature according to the skeleton point information;

[0140] The third module 103 is used to perform denoising on the first surface electromyography signal by threshold filtering to obtain a second surface electromyography signal;

[0141] The fourth module 104 is used to extract the joint distance feature, the slow motion feature, the fast motion feature and the feature map of the second surface electromyography signal through an embedding operation, and obtain a joint distance feature map, a slow motion feature map, a fast motion feature map and a surface electromyography signal feature map accordingly;

[0142] The fifth module 105 is used to perform feature fusion processing on the joint distance feature map, the slow motion feature map, the fast motion feature map and the surface electromyography signal feature map to obtain a fusion feature map;

[0143] The sixth module 106 is used to obtain a gesture recognition result according to the fused feature map.

[0144] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0145] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned multimodal gesture recognition method based on myoelectric and visual fusion when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0146] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0147] See also Figure 6 , Figure 6 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0148] The processor 201 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0149] The memory 202 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 202 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 202, and the processor 201 calls and executes a multimodal gesture recognition method based on electromyography and visual fusion in the embodiment of this application;

[0150] Input / output interface 203, used to implement information input and output;

[0151] Communication interface 204, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0152] Bus 205 , which transmits information between various components of the device (e.g., processor 201 , memory 202 , input / output interface 203 , and communication interface 204 );

[0153] The processor 201 , the memory 202 , the input / output interface 203 and the communication interface 204 are connected to each other in communication within the device via the bus 205 .

[0154] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned multimodal gesture recognition method based on electromyography and vision fusion.

[0155] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0156] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0157] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0158] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0159] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0160] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0161] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0162] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0163] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0164] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0166] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0167] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A multimodal gesture recognition method based on electromyography and vision fusion, characterized in that: The following steps are involved: Obtaining skeleton point information and first surface electromyography signal of the hand image; Calculate joint distance features, slow motion features and fast motion features according to the skeleton point information; Performing denoising processing on the first surface electromyography signal by threshold filtering to obtain a second surface electromyography signal; Extracting the joint distance feature, the slow motion feature, the fast motion feature and the feature map of the second surface electromyography signal through an embedding operation, and obtaining a joint distance feature map, a slow motion feature map, a fast motion feature map and a surface electromyography signal feature map accordingly; Performing feature fusion processing on the joint joint distance feature map, the slow motion feature map, the fast motion feature map and the surface electromyography signal feature map to obtain a fused feature map; Obtaining a gesture recognition result according to the fused feature map; The step of acquiring the skeleton point information and the first surface electromyography signal of the hand image comprises the following steps: Acquire a hand video; wherein the hand video includes several frames of hand images; Using a hand gesture and key point recognition algorithm to identify skeleton point information based on the hand image; Collecting a first surface electromyography signal of the hand through a first device; The step of calculating the joint distance feature, the slow motion feature and the fast motion feature according to the skeleton point information comprises the following steps: Determine the Cartesian coordinates of each joint according to the skeleton point information; Determine a joint distance feature according to the number of frames, the number of joint points and the Cartesian coordinates; Determine a slow motion feature according to current skeleton point information of a current frame and first skeleton point information of a first adjacent frame; wherein the first adjacent frame is directly adjacent to the current frame; Determine the fast motion feature according to the current skeleton point information of the current frame and the second skeleton point information of the second adjacent frame; wherein the second adjacent frame is one frame apart from the current frame; The calculation formula of the joint distance feature is: ; The expression of the slow motion feature is: ; The expression of the fast motion feature is: ; in, express The joint distance feature of Represents the skeleton point information of the kth frame; express and The Euclidean distance between represents the slow motion of the kth frame; Indicates fast motion in the kth frame.

2. The method according to claim 1, characterized in that The method of performing denoising processing on the first surface electromyography signal by threshold filtering to obtain a second surface electromyography signal comprises the following steps: Performing discrete wavelet transform processing on the first surface electromyography signal to obtain a first processing result; Performing threshold processing on the first processing result to obtain a second processing result; The second processing result is subjected to inverse discrete wavelet transform processing to obtain a second surface electromyography signal.

3. The method according to claim 1, characterized in that Obtaining a gesture recognition result according to the fused feature map comprises the following steps: Using a one-dimensional convolution block to extract features from the fused feature map to obtain a third processing result; Performing a first activation operation and a batch normalization operation on the third processing result to obtain a fourth processing result; Performing global average pooling processing on the fourth processing result to obtain a fifth processing result; Performing full connection processing on the fifth processing result to obtain a sixth processing result; A second activation operation is performed on the sixth processing result to obtain a gesture recognition result.

4. The method according to claim 1, characterized in that: The embedding operation uses a convolution block and / or a pooling layer, and the expression of the convolution block is: ; in, Represents a one-dimensional convolution kernel with a convolution kernel size of 1 and a number of channels. ; Indicates the filter value.

5. The method according to claim 1, characterized in that The expression of the feature fusion processing is: ; in, represents the k-th frame fusion feature map, represents a fusion operation; Indicates that the kth frame is embedded by the operation The extracted joint distance features; Indicates that the kth frame is embedded by the operation The extracted slow motion features; Indicates that the kth frame is embedded by the operation The extracted fast motion features; Indicates that the kth frame is embedded by the operation Extracted surface electromyography signal features.

6. A multimodal gesture recognition system based on electromyography and vision fusion, characterized in that: include: The first module is used to obtain the skeleton point information and the first surface electromyography signal of the hand image; The second module is used to calculate the joint distance feature, the slow motion feature and the fast motion feature according to the skeleton point information; A third module is used to perform denoising on the first surface electromyography signal by threshold filtering to obtain a second surface electromyography signal; A fourth module is used to extract the joint distance feature, the slow motion feature, the fast motion feature and the feature map of the second surface electromyography signal through an embedding operation, and obtain a joint distance feature map, a slow motion feature map, a fast motion feature map and a surface electromyography signal feature map accordingly; A fifth module is used to perform feature fusion processing on the joint distance feature map, the slow motion feature map, the fast motion feature map and the surface electromyography signal feature map to obtain a fusion feature map; A sixth module is used to obtain a gesture recognition result according to the fusion feature map; The step of acquiring the skeleton point information and the first surface electromyography signal of the hand image comprises the following steps: Acquire a hand video; wherein the hand video includes several frames of hand images; Using a hand gesture and key point recognition algorithm to identify skeleton point information based on the hand image; Collecting a first surface electromyography signal of the hand through a first device; The step of calculating the joint distance feature, the slow motion feature and the fast motion feature according to the skeleton point information comprises the following steps: Determine the Cartesian coordinates of each joint according to the skeleton point information; Determine a joint distance feature according to the number of frames, the number of joint points and the Cartesian coordinates; Determine a slow motion feature according to current skeleton point information of a current frame and first skeleton point information of a first adjacent frame; wherein the first adjacent frame is directly adjacent to the current frame; Determine the fast motion feature according to the current skeleton point information of the current frame and the second skeleton point information of the second adjacent frame; wherein the second adjacent frame is one frame apart from the current frame; The calculation formula of the joint distance feature is: ; The expression of the slow motion feature is: ; The expression of the fast motion feature is: ; in, express The joint distance feature of Represents the skeleton point information of the kth frame; express and The Euclidean distance between represents the slow motion of the kth frame; Indicates fast motion in the kth frame.

7. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 5.

8. A computer storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the method according to any one of claims 1 to 5 when executed by the processor.