Gesture recognition method and gesture recognition system based on multi-modal feature fusion

Abstract

The invention relates to a gesture recognition method and a gesture recognition system based on multi-modal feature fusion, and the method comprises the steps: constructing an improved multi-modal feature fusion network, obtaining historical multi-modal features at corresponding moments, and inputting the improved multi-modal feature fusion network for training; obtaining multi-modal features at corresponding moments, inputting the multi-modal features into the trained network, and obtaining a gesture recognition result; according to the system, an image containing a hand is obtained through an image sensor, an electromyographic signal of the hand is obtained through hand wearing equipment provided with an electromyographic sensor, environment brightness information is obtained through an environment illumination sensor, and a controller is configured to execute the method during gesture recognition. According to the method, the myoelectricity features provided by the myoelectricity sensor are acquired, the myoelectricity features are fused with the image features in a multi-mode mode, the gesture features are acquired in a multi-angle mode, the problems that the external environment is sensitive and the hand is prone to being shielded during single image recognition are solved, the defects of single-mode gesture recognition are overcome, the gesture recognition accuracy is improved, and the user experience is improved. Particularly, the accuracy under the shielding condition is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, and in particular to a gesture recognition method and a gesture recognition system based on multimodal feature fusion. Background Art

[0002] With the rapid development of science and technology, technologies such as virtual reality and mixed reality are gradually becoming bridges connecting the physical world and the digital world. Gesture interaction is an important part of realizing this technology. With the continuous development of virtual reality and mixed reality technologies, the application scenarios of gesture interaction are also gradually expanding and better changing the user's interactive experience.

[0003] In the field of virtual reality, gesture interaction refers to a way of human-computer interaction through hand movements, which can enable users to operate the device more intuitively, improve user experience, and expand the interactive dimension of the device. In order to improve the operability of gesture interaction, it is necessary to improve the ability of gesture recognition and its success rate.

[0004] In the prior art, the mainstream gesture recognition methods are generally based on cameras and perform gesture recognition by extracting image features, including 2D gesture recognition and 3D gesture recognition. In 2D gesture recognition, the model extracts a single camera image feature and outputs the gesture category through a gesture classifier, while in 3D gesture recognition, the model extracts two or more camera image features, outputs the 3D key point positions of the hand, and then obtains the gesture category through post-processing logic.

[0005] The camera-based gesture recognition method has the advantages of low cost and wide application range, but it also has shortcomings. The main reason is that the camera has its own observation angle. When one hand moves to a certain angle or both hands move interactively, occlusion will occur, causing the hand image features to be lost, thereby affecting the accuracy of gesture recognition. In addition, the image features of the camera are easily affected by the external lighting environment, which can easily lead to inaccurate gesture recognition. Summary of the invention

[0006] The present invention solves the problems existing in the prior art and provides a gesture recognition method and a gesture recognition system based on multimodal feature fusion.

[0007] The technical solution adopted by the present invention is a gesture recognition method based on multimodal feature fusion, which comprises the following steps:

[0008] S1 builds an improved multimodal feature fusion network;

[0009] S2 obtains historical multimodal features at the corresponding moment and inputs them into the improved multimodal feature fusion network for training;

[0010] S3 obtains the multimodal features at the corresponding moment, inputs them into the trained network, and obtains the gesture recognition result.

[0011] Preferably, the multimodal feature fusion network includes a multimodal feature extraction input module, a feature concatenation layer, a fully connected layer and a classifier arranged in sequence.

[0012] Preferably, the multimodal feature extraction input module includes a parallel image feature extraction input branch and an electromyographic feature extraction input branch.

[0013] Preferably, extracting features using the image feature extraction input branch includes the following steps:

[0014] S1.1.1 obtain an original image with a background, extract a hand contour, and obtain a hand image;

[0015] S1.1.2 Obtain the left and right attributes of the hand based on the camera rules;

[0016] S1.1.3 extract the positions of 21 3D key points of the hand;

[0017] S1.1.4 constructs image features based on the results of S1.2 and S1.3.

[0018] Preferably, extracting features by extracting input branches with electromyographic features comprises the following steps:

[0019] S1.2.1 Collect and pre-process the electromyographic signal data of the hand-worn device;

[0020] S1.2.2 extracts the electromyographic features of the preprocessed electromyographic signal data using a deep autoencoder network.

[0021] Preferably, a cross attention module is provided between the multimodal feature extraction input module and the feature splicing layer, and the cross attention module is then connected to the feature splicing layer through a pooling layer.

[0022] Preferably, a feature enhancement module cooperating with a temporal credible weight module is provided in conjunction with the cross-attention module, and the temporal credible weight module is configured in a multimodal feature extraction input module.

[0023] Preferably, a weight control layer is provided between the feature concatenation layer and the fully connected layer, and the weight control layer is provided in coordination with the output end of the classifier.

[0024] Preferably, the weight control layer is associated with image integrity, ambient brightness and a network loss function.

[0025] A multi-modal feature fusion gesture recognition system, the system comprising:

[0026] one or more image sensors for acquiring an image containing a hand;

[0027] One or more hand-worn devices, equipped with electromyographic sensors, for acquiring electromyographic signals of the hands;

[0028] One or more ambient illumination sensors, used to obtain ambient brightness information;

[0029] A controller is provided in conjunction with the image sensor, the electromyography sensor and the ambient illumination sensor, and is used to execute the gesture recognition method based on multimodal feature fusion during gesture recognition.

[0030] The present invention relates to a gesture recognition method and a gesture recognition system based on multimodal feature fusion, which construct an improved multimodal feature fusion network, obtain historical multimodal features at corresponding moments, and input the improved multimodal feature fusion network for training; obtain multimodal features at corresponding moments, input the trained network, and obtain gesture recognition results; the system uses an image sensor to obtain an image containing a hand, uses a hand wearable device equipped with an electromyographic sensor to obtain an electromyographic signal of the hand, uses an ambient illumination sensor to obtain ambient brightness information, and configures a controller to execute the method during gesture recognition.

[0031] The beneficial effects of the present invention are that by acquiring electromyographic features including but not limited to those provided by electromyographic sensors, performing multimodal fusion with image features, and acquiring gesture features from multiple angles, the problems of sensitivity to the external environment and easy occlusion of hands in single image recognition are solved, the shortcomings of single-modal gesture recognition are overcome, and the accuracy of gesture recognition is improved, especially the accuracy of gesture recognition under occlusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flow chart of the method of the present invention;

[0033] Figure 2 A schematic diagram of the network structure applied in the present invention;

[0034] Figure 3 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0035] The present invention is further described in detail below in conjunction with embodiments, but the protection scope of the present invention is not limited thereto.

[0036] The present invention relates to a multi-modal feature fusion gesture recognition system, the system comprising:

[0037] one or more image sensors for acquiring an image containing a hand;

[0038] One or more hand-worn devices, equipped with electromyographic sensors, for acquiring electromyographic signals of the hands;

[0039] One or more ambient illumination sensors, used to obtain ambient brightness information;

[0040] A controller is provided in conjunction with the image sensor, the electromyography sensor and the ambient illumination sensor, and is used to execute the gesture recognition method based on multimodal feature fusion during gesture recognition.

[0041] In the present invention, a gesture recognition system based on multimodal feature fusion uses an image sensor, generally a multi-eye camera, to obtain hand images for detecting hand features and extracting hand key points. At the same time, a hand wearable device is configured in the system, which is worn on the hand of the user. Its built-in electromyography sensor obtains the electromyography signal consistent with the image time, and completes the multimodal feature fusion recognition after extracting the electromyography features.

[0042] In the present invention, the hand-worn device can be hidden in some scenarios, and the electromyographic sensor can be directly configured with a gel patch and other structures to achieve close contact with the user's hand and obtain electromyographic signals.

[0043] In the present invention, taking into account the sensitivity of the image sensor to the ambient illumination, the ambient illumination sensor is used to obtain the ambient brightness information, and the confidence of the hand features obtained by the image sensor is adjusted based on the ambient brightness, so that the improved multimodal feature fusion network can be better utilized to obtain the fusion weights of the multimodal features, thereby obtaining more accurate gesture recognition results.

[0044] In the system, the controller executes the gesture recognition method based on multimodal feature fusion during gesture recognition, and the method includes the following steps:

[0045] S1 builds an improved multimodal feature fusion network;

[0046] S2 obtains historical multimodal features at the corresponding moment and inputs them into the improved multimodal feature fusion network for training;

[0047] S3 obtains the multimodal features at the corresponding moment, inputs them into the trained network, and obtains the gesture recognition result.

[0048] In the present invention, the improved multimodal feature fusion network is the core of method identification, and the multimodal feature fusion network includes a multimodal feature extraction input module, a feature splicing layer, a fully connected layer and a classifier arranged in sequence;

[0049] Among them, the multimodal features are extracted and regularized by the multimodal feature extraction input module, and the multimodal features are aligned and spliced ​​by the feature splicing layer. The alignment here includes but is not limited to the temporal alignment of the features at the same moment to ensure more accurate feature fusion. The learned feature representation is mapped to the label space by the fully connected layer, and the classification output is realized through the classifier.

[0050] In the improved multimodal feature fusion network, the multimodal feature extraction input module is the core, which includes a parallel image feature extraction input branch and an electromyography feature extraction input branch.

[0051] Extracting features using the image feature extraction input branch includes the following steps:

[0052] S1.1.1 obtain an original image with a background, extract a hand contour, and obtain a hand image;

[0053] S1.1.2 Obtain the left and right attributes of the hand based on the camera rules;

[0054] S1.1.3 extract the positions of 21 3D key points of the hand;

[0055] S1.1.4 constructs image features based on the results of S1.2 and S1.3.

[0056] In the present invention, after collecting the original image with background, it is necessary to extract the hand contour. This extraction of the hand contour can be obtained by establishing a skin color model, and then through color recognition of the feature position and hand contour learning. This is content that can be easily understood by technical personnel in the field. At the same time, according to camera rules, such as whether it is a mirror image or not, the obtained hand contour is used to determine whether it is the user's left hand or right hand, establish a spatial coordinate system, and extract the positions of 21 3D hand key points. The 21 3D key points are 21 joints of the hand skeleton, which are well known to technical personnel in the field. After completion, the image features of the hand key points can be constructed.

[0057] Extracting features from the input branch using electromyographic features includes the following steps:

[0058] S1.2.1 Collect and pre-process the electromyographic signal data of the hand-worn device;

[0059] S1.2.2 extracts the electromyographic features of the preprocessed electromyographic signal data using a deep autoencoder network.

[0060] In the present invention, the electromyographic signal data of the hand wearable device is collected and preprocessed, including but not limited to eliminating noise and interference, removing invalid signals, etc., and then extracting time domain and frequency domain features, and then performing end-to-end feature learning and extraction on the original electromyographic signal with a deep autoencoder network to obtain feature expression;

[0061] It should be noted that the obtained electromyographic features can be classified and identified using classifiers including but not limited to SVM, KNN, etc.

[0062] In the present invention, the multimodal feature fusion network is further improved so that it can better balance the fusion weights of image features and electromyographic features, reduce the amount of calculation, and be applicable to more scenarios to complement each other;

[0063] First, a cross-attention module and a pooling layer are set between the multimodal feature extraction input module and the feature concatenation layer. The cross-attention module is used to weight the feature space, extract important features from image features and electromyographic features, and assign appropriate weights to improve network performance. In particular, in multimodal processing, the matching between different modalities can be complemented and enhanced, and the weights of different patches are clearly displayed in the pooling layer, which enhances the network interpretability and improves the performance of cross-modal tasks.

[0064] Secondly, a temporal credibility weight module is configured to detect the time delay of two feature input branches. For a branch with a larger time delay, its temporal credibility is low. Therefore, when the cross-attention module processes the features, the feature enhancement module is used to enhance the features with higher temporal credibility.

[0065] Finally, a weight control layer is set between the feature concatenation layer and the fully connected layer to optimize the training parameters by calculating the loss.

[0066] Specifically, the weight control at the weight control layer is related to the loss function L, image completeness, and ambient brightness. The higher the image completeness, the greater the feature weight of the image feature extraction input branch at the fully connected layer. When the ambient brightness is within the preset threshold, the greater the feature weight of the electromyography feature extraction input branch at the fully connected layer. At the same time, the specific weight ratio is controlled by the loss function.

[0067] In the present invention, different loss functions can be set based on different training focuses. Here, the loss function L satisfies:

[0068]

[0069] Where N is the number of gesture categories, is the probability of predicting the nth class, K is the number of key points, The predicted position of the kth key point; y n is the true value of the gesture category, p k is the true value of the key point position.

[0070] An embodiment of the present invention in application is given below.

[0071] The image sensor obtains the original image, obtains the left and right hand category information and the hand position information through hand detection, and cuts the hand image from the original image according to the hand position information;

[0072] The hand image is input into the image feature extraction input branch, and the positions of 21 3D key points of the hand are obtained using a multi-layer convolutional neural network;

[0073] The electromyographic sensor collects electromyographic data at the wrist, uses a bandpass filter to filter the electromyographic data, and then performs noise reduction processing on the filtered data to obtain a noise-reduced electromyographic signal;

[0074] The electromyographic signal is input into the electromyographic feature extraction input branch, and the electromyographic information is processed using a deep autoencoder network to obtain electromyographic features;

[0075] The positions of 21 3D key points of the hand are input into the improved multimodal feature fusion network with the above-mentioned electromyographic features. After feature splicing, the spliced ​​features are processed by a fully connected layer and the gesture category is output by a classifier.

[0076] The present invention also relates to a computer-readable storage medium in its application, on which a gesture recognition program based on multimodal feature fusion is stored, and when the program is executed by a processor, the gesture recognition method based on multimodal feature fusion is implemented.

[0077] The present invention also relates to a computer device in application, including a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the above-mentioned gesture recognition method based on multimodal feature fusion is implemented.

[0078] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0079] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0080] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0081] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0082] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0083] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A gesture recognition method based on multimodal feature fusion, characterized in that: The method comprises the following steps: S1 builds an improved multimodal feature fusion network; S2 obtains historical multimodal features at the corresponding moment and inputs them into the improved multimodal feature fusion network for training; S3 obtains the multimodal features at the corresponding moment, inputs them into the trained network, and obtains the gesture recognition results.

2. The gesture recognition method based on multimodal feature fusion according to claim 1, characterized in that: The multimodal feature fusion network includes a multimodal feature extraction input module, a feature splicing layer, a fully connected layer and a classifier which are arranged in sequence.

3. The gesture recognition method based on multimodal feature fusion according to claim 2, characterized in that: The multimodal feature extraction input module includes a parallel image feature extraction input branch and an electromyographic feature extraction input branch.

4. The gesture recognition method based on multimodal feature fusion according to claim 3, characterized in that: Extracting features using the image feature extraction input branch includes the following steps: S1.1.1 Obtain an original image with a background, extract a hand outline, and obtain a hand image; S1.1.2 Obtain the left and right attributes of the hand based on camera rules; S1.1.3 Extract the positions of 21 3D key points of the hand; S1.1.4 Construct image features based on the results of S1.2 and S1.

3.

5. The gesture recognition method based on multimodal feature fusion according to claim 3, characterized in that: Extracting features from the input branch using electromyographic features includes the following steps: S1.2.1 Collect and pre-process the electromyographic signal data of the hand-worn device; S1.2.2 Extract the electromyographic features of the preprocessed electromyographic signal data using a deep autoencoder network.

6. The gesture recognition method based on multimodal feature fusion according to claim 2, characterized in that: A cross attention module is provided between the multimodal feature extraction input module and the feature splicing layer, and the cross attention module is connected to the feature splicing layer through a pooling layer.

7. The method for gesture recognition based on multimodal feature fusion according to claim 6, characterized in that: A feature enhancement module is provided in conjunction with the cross-attention module and is coordinated with the temporal credible weight module, and the temporal credible weight module is configured in the multimodal feature extraction input module.

8. The gesture recognition method based on multimodal feature fusion according to claim 2, characterized in that: A weight control layer is provided between the feature concatenation layer and the fully connected layer, and the weight control layer is provided in coordination with the output end of the classifier.

9. The method for gesture recognition based on multimodal feature fusion according to claim 8, characterized in that: The weight control layer is associated with image integrity, ambient brightness and the network's loss function.

10. A multimodal feature fusion gesture recognition system, characterized in that: The system comprises: one or more image sensors for acquiring an image containing a hand; One or more hand-worn devices, equipped with electromyographic sensors, for acquiring electromyographic signals of the hands; One or more ambient light sensors for obtaining ambient brightness information; A controller is provided in conjunction with the image sensor, the electromyography sensor and the ambient light sensor, for executing the gesture recognition method based on multimodal feature fusion as described in any one of claims 1 to 9 during gesture recognition.

Images