Gesture recognition methods, devices, electronic devices, and storage media

Abstract

This disclosure provides a gesture recognition method, apparatus, electronic device, and storage medium, relating to the field of data processing technology, particularly to the fields of artificial intelligence, deep learning, autonomous driving, virtual reality, and human-computer interaction. The specific implementation scheme is as follows: based on a video of a dynamic gesture to be recognized, multimodal information of the dynamic gesture to be recognized is acquired; the multimodal information is fused to obtain fused information of the dynamic gesture to be recognized; based on the fused information, the gesture features of the dynamic gesture to be recognized are determined; and based on the gesture features, the gesture category of the dynamic gesture to be recognized is determined. The embodiments of this disclosure can accurately and quickly recognize dynamic gestures.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to the fields of artificial intelligence, deep learning, autonomous driving, virtual reality, and human-computer interaction. Background Technology

[0002] With the development of technology, human-computer interaction has become increasingly convenient and simple. Gesture interaction is an important aspect of human-computer interaction. The key to gesture interaction lies in accurately recognizing gestures. Gesture recognition includes recognizing static and dynamic gestures. Compared to static gestures, dynamic gestures are closer to human expression habits and have more practical significance. Therefore, accurately understanding dynamic gestures is particularly crucial. Summary of the Invention

[0003] This disclosure provides gesture recognition methods, devices, electronic devices, and storage media.

[0004] According to one aspect of this disclosure, a gesture recognition method is provided, comprising:

[0005] Based on the video of the dynamic gesture to be recognized, obtain the multimodal information of the dynamic gesture to be recognized;

[0006] Multimodal information is fused to obtain fused information of the dynamic gesture to be recognized;

[0007] Based on the fused information, the gesture features of the dynamic gesture to be recognized are determined;

[0008] The gesture category of the dynamic gesture to be recognized is determined based on gesture features.

[0009] According to another aspect of this disclosure, a gesture recognition device is provided, comprising:

[0010] The information acquisition module is used to acquire multimodal information of the dynamic gesture to be recognized based on the video of the dynamic gesture to be recognized;

[0011] The fusion module is used to fuse multimodal information to obtain fused information of the dynamic gesture to be recognized;

[0012] The feature extraction module is used to determine the gesture features of the dynamic gesture to be recognized based on the fused information;

[0013] The recognition module is used to determine the gesture category of the dynamic gesture to be recognized based on gesture features.

[0014] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0015] At least one processor; and

[0016] The memory is communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods of any embodiment of the present disclosure.

[0018] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform a method according to any embodiment of this disclosure.

[0019] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a method according to any embodiment of this disclosure.

[0020] The embodiments disclosed herein can accurately and efficiently recognize dynamic gestures.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0022] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0023] Figure 1 This is a flowchart illustrating the gesture recognition method provided according to an embodiment of the present disclosure;

[0024] Figure 2 This is another schematic flowchart of the gesture recognition method provided according to an embodiment of the present disclosure;

[0025] Figure 3 This is a schematic diagram of the structure of the recognition model provided according to the embodiments of this disclosure;

[0026] Figure 4 This is a flowchart illustrating the training of a recognition model according to an embodiment of the present disclosure;

[0027] Figure 5 This is a flowchart illustrating the gesture recognition method provided according to an embodiment of the present disclosure;

[0028] Figure 6 This is a schematic diagram of the structure of a gesture recognition device according to an embodiment of the present disclosure;

[0029] Figure 7 This is a block diagram of an electronic device used to implement the gesture recognition method of the embodiments of this disclosure. Detailed Implementation

[0030] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0031] With the rapid development of technology, artificial intelligence is booming, and human-computer interaction is gradually becoming an indispensable part of life and work. Gestures are a natural form of human communication and can convey rich semantic information. Therefore, gesture recognition is a key research area in the field of human-computer interaction and has been widely applied in virtual reality, smart homes, and other fields.

[0032] However, gestures themselves are flexible, unpredictable, and diverse, making gesture recognition a challenging multidisciplinary research area.

[0033] Gesture recognition can be divided into static gesture recognition and dynamic gesture recognition. Static gesture recognition focuses only on the shape features at a single point in time, while dynamic gesture recognition focuses on actions over a period of time, adding time information and action features. Compared to static gesture recognition, dynamic gesture recognition is closer to human expression habits and has greater practical significance. Therefore, accurately understanding dynamic gestures plays a decisive role in the accuracy of human-computer interaction.

[0034] Furthermore, gesture recognition is applied in multiple fields, and there are certain requirements for the timeliness of gesture recognition. To accurately and efficiently understand dynamic gestures, embodiments of this disclosure provide a gesture recognition method, such as... Figure 1 The diagram shown is a flowchart of the method, including:

[0035] S101, Based on the video of the dynamic gesture to be recognized, obtain the multimodal information of the dynamic gesture to be recognized.

[0036] Multimodal information, as the name suggests, is information from multiple modalities. Each modal information contains a description of the dynamic gesture to be recognized, which can cover multiple aspects such as shape, distance, gesture changes, texture, and color.

[0037] Each modality contains a large amount of description of the dynamic gesture to be recognized. In order to avoid the key information being weakened by complex processing, in this embodiment of the present disclosure, the multimodal information is fused in S102 to obtain the fused information of the dynamic gesture to be recognized.

[0038] That is, in this embodiment of the present disclosure, the original multimodal information is first fused. On the one hand, this allows as many descriptions contained in the original multimodal information as possible to be retained in the fused information to improve the accuracy of dynamic gesture recognition. On the other hand, fusing the multimodal information first also makes it easier to use a lightweight neural network to extract gesture features.

[0039] In S103, the gesture features of the dynamic gesture to be recognized are determined based on the fused information.

[0040] S104, determine the gesture category of the dynamic gesture to be recognized based on gesture features.

[0041] During implementation, the gesture features can be classified to obtain the probability distribution of the dynamic gesture to be recognized in each gesture category. When the maximum probability value in the probability distribution is greater than a preset threshold, the gesture category corresponding to the maximum probability value can be determined as the category of the dynamic gesture to be recognized.

[0042] In this embodiment, multimodal information of the dynamic gesture to be recognized is first acquired. This multimodal information contains a large amount of original information describing the dynamic gesture. By fusing the multimodal information, the original information of the dynamic gesture to be recognized can be preserved to the greatest extent, so that key features can be extracted during subsequent gesture feature extraction, thereby improving the accuracy of dynamic gesture recognition. Furthermore, fusing multimodal information before extracting gesture features simplifies the neural network for gesture feature extraction, making the neural network lightweight and compatible with various electronic devices with varying performance levels, thus increasing the applicability of dynamic gesture recognition. In addition, a lightweight neural network implies a simpler processing method, thereby improving the efficiency of dynamic gesture recognition. In short, this embodiment provides a universal, accurate, and efficient method for dynamic gesture recognition.

[0043] In some embodiments, multimodal information may include at least one of the following: color image information, depth information, and optical flow information.

[0044] The color image information can be an RGB (red, green, blue) image from the video. The color image information includes the gesture foreground and other content as the background. Based on the color image information, dynamic gestures can be well recognized.

[0045] Depth information refers to the depth map. In practice, an RGB-D (RGB Depth) map can be captured from the gesture, thereby obtaining color image information and depth information.

[0046] Color image information is easily affected by lighting, skin color, and complex backgrounds, so depth information can be incorporated to improve the accuracy of dynamic gesture recognition.

[0047] However, dynamic gestures contain changes in movement, i.e., motion information. Color image information and depth information are not direct enough in describing the changes in gestures. Therefore, in this embodiment, motion information can also be extracted to improve the accuracy of dynamic gesture recognition.

[0048] Optical flow represents the instantaneous velocity of each pixel of a 3D object moving across the image plane. Generally, optical flow is also a microcosm of the object's motion changes between two adjacent frames. From optical flow information, we can obtain not only the object's direction and speed of motion, but also its distance and angle. Therefore, optical flow information can effectively represent the object's motion process. Moreover, the extraction of optical flow information is relatively simple and efficient. Therefore, in this embodiment, optical flow information is used to describe the changes in dynamic gestures to improve the accuracy of dynamic gesture recognition while also considering the timeliness of dynamic gesture recognition.

[0049] In this embodiment, the optical flow map of each frame can be extracted and saved using the calcOpticalFlowPyrLK function in OpenCV based on the Lucas-Kanade method.

[0050] In some embodiments, excessively long video content results in a large amount of data to be processed, increasing the time required for dynamic gesture recognition and making it difficult to guarantee the timeliness of dynamic gesture recognition. For excessively long videos, certain methods are needed to obtain key video content to improve processing efficiency.

[0051] However, videos that are too short are not conducive to accurately recognizing dynamic gestures. Therefore, it is necessary to interpolate frames from excessively short videos to improve the accuracy of dynamic gesture recognition.

[0052] In summary, to balance the timeliness and accuracy of dynamic gesture recognition, in this embodiment of the disclosure, obtaining multimodal information of the dynamic gesture to be recognized based on the video of the gesture can be implemented as follows:

[0053] Step A1: The number of frames in the video is processed to the target number of frames by sampling or interpolation methods to obtain the image sequence of the dynamic gesture to be recognized.

[0054] For excessively long videos, downsampling can reduce the number of video frames, improving the timeliness of dynamic gesture recognition. Downsampling can be performed using either equal-interval or non-equal-interval sampling. Furthermore, dynamic gestures generally don't occupy excessively long durations; the beginning and end of a video often contain content unrelated to the gesture being recognized. Therefore, to ensure accuracy, non-equal-interval sampling can be implemented by trimming the video frames at both ends, retaining only the key frames relevant to the dynamic gesture. Similarly, sampling can also involve analyzing the video content to identify keyframes. The required image sequence is then constructed using these keyframes. Compared to keyframe analysis, non-equal-interval sampling offers better timeliness; the appropriate sampling method can be chosen based on actual needs.

[0055] For excessively short videos, interpolation can be used to increase the video length. Interpolation can generate intermediate frames based on multiple adjacent video frames. Alternatively, all or some video frames can be repeated to increase the video length.

[0056] To improve processing efficiency while maintaining accuracy, image sequences can be cropped to a specified size during implementation. For example, each frame in the image sequence can be processed to 112*112 pixels. Then, multimodal information can be extracted. Of course, optical flow information can be extracted from video frames downsized to 112*112 pixels, or from the unscaled image sequence.

[0057] Step A2: Extract multimodal information from the image sequence.

[0058] In this embodiment of the disclosure, an image sequence with the required number of frames is generated based on the video to ensure that the image sequence contains enough information for accurate dynamic gesture recognition. Furthermore, the length of the image sequence can limit the amount of information processed subsequently, thereby improving the efficiency of dynamic gesture recognition.

[0059] The optical flow information in an image sequence includes multiple frames of optical flow maps. However, not every frame of optical flow map is equally important for recognizing dynamic gestures. Therefore, to improve the accuracy of dynamic gesture recognition, it can be implemented as follows: Figure 2 As shown:

[0060] S201, divide the image sequence into n parts to obtain n subsequences, where n is a positive integer greater than 1.

[0061] S202, for each subsequence, perform the following: determine the importance of the subsequence and use the importance as the weight of the optical flow graph of the subsequence.

[0062] This disclosure provides, for example, two methods for determining the importance of subsequences (i.e., the weights in the optical flow graph):

[0063] Option 1) includes the following steps B1-B2:

[0064] Step B1, determine at least one of the following importance metrics: image quality of the subsequence, gesture integrity, and the degree of data difference within the multi-frame optical flow map of the subsequence.

[0065] Image quality can be determined using an image quality assessment model for each subsequence. Alternatively, image quality can be determined based on a combination of factors, including contrast, dynamic range of brightness, and sharpness. Higher image quality results in more accurate extraction of optical flow information, which is more beneficial for dynamic gesture recognition.

[0066] Gesture edges can be extracted from each frame of a subsequence using edge operators to determine whether the included gesture is complete. This yields the gesture completeness score. The more complete the gesture, the more accurate the semantics, and the better it is for dynamic gesture recognition.

[0067] Furthermore, if the differences between the optical flow maps of each frame in a subsequence are too small, it indicates that the gesture changes are small, and the motion information of this subsequence has little impact on the accuracy of gesture recognition. If the differences between the optical flow maps of each frame in a subsequence are large, then the subsequence contains rich motion information and has a greater impact on the accuracy of gesture recognition.

[0068] Step B2: Determine the importance of the subsequences based on the positive correlation between each importance indicator and the degree of importance.

[0069] The importance index of the embodiments disclosed herein can reasonably measure the importance of subsequences to dynamic gesture recognition.

[0070] Option 2) includes the following steps C1-C2:

[0071] Step C1: Determine the mean optical flow of the optical flow map of the subsequence.

[0072] Step C2: The ratio between the mean optical flow and the mean optical flow of the image sequence is used to determine the importance of the subsequence.

[0073] For example, the mean optical flow of the optical flow map of subsequence 1 is G1, and the mean optical flow of the optical flow map of subsequence 2 is G2. The mean optical flow of the optical flow map of the image sequence is G. Then the importance of subsequence 1 is (G1 / G), and similarly, the importance of subsequence 2 is (G2 / G).

[0074] In this embodiment, the importance of each frame's optical flow map can be directly determined based on the data distribution of the optical flow data itself. This method can efficiently determine the importance of subsequences, laying a solid data foundation for the weighting of optical flow.

[0075] S203, the optical flow maps of each subsequence in the original optical flow map of the image sequence are weighted and summed to obtain the optical flow information of the dynamic gesture to be recognized.

[0076] In this embodiment of the disclosure, by determining the importance of sub-sequences, optical flow data that is important to the dynamic gesture to be recognized is selected, while secondary optical flow data is weakened, so as to accurately extract gesture features and improve the accuracy of dynamic gesture recognition.

[0077] Gesture feature extraction based on image sequence fusion information can be implemented as follows: extract short-term spatiotemporal features of the dynamic gesture to be recognized from the fusion information; encode the short-term spatiotemporal features into long-term spatiotemporal features. Short-term spatiotemporal features can be extracted using 3D residual networks or Temporal 3D ConvNets (T3D) networks. The short-term spatiotemporal features are then input into a Temporal Convolutional Network (TCN), which encodes the short-term spatiotemporal features to obtain the long-term spatiotemporal features.

[0078] In this embodiment of the disclosure, in order to reasonably fuse multimodal information, a self-attention mechanism can be used to perform nonlinear fusion of multimodal information to obtain fused information of the dynamic gesture to be recognized.

[0079] Nonlinear fusion of multimodal information based on self-attention mechanism can adaptively fuse different modal information for different dynamic gestures, so as to obtain fused information suitable for different dynamic gestures and improve the recognition accuracy of dynamic gestures.

[0080] The process of fusing multimodal information can be implemented as follows:

[0081] Step D1: Determine the weights of each piece of information in the multimodal information based on the self-attention mechanism.

[0082] Step D2: Based on the weights of each piece of information, the multimodal information is weighted and summed to obtain the fused information of the dynamic gesture to be recognized.

[0083] In this embodiment of the disclosure, a weighted fusion of multimodal information is achieved based on a self-attention mechanism. This enables the selection of information required for the dynamic selection of different dynamic gestures throughout the feature extraction process, so as to adaptively fuse different dynamic gestures and improve the accuracy of dynamic gesture recognition.

[0084] Assume the RGB image sequence (i.e., color image information) is S rgb The depth image sequence (i.e., depth information) is S depth Optical flow information is S flow The image sequence after the fusion of the three modalities is z (i.e., fused information). The weighted sum of the information from the three modalities is shown in equation (1):

[0085]

[0086] Where α=[α rgb α depth α flow [ ] represents the weighting coefficient. In one possible implementation, the weighting coefficient is calculated as shown in equation (2):

[0087]

[0088] In formula (2), Indicates S rgb Perform average pooling operation. Indicates S depth Perform average pooling operation. Indicates S flow Perform average pooling operation, W 01 Indicates the corresponding S rgb The convolution weights W of the convolutional layer 02 Indicates the corresponding S depth The convolution weights W of the convolutional layer 03 Indicates the corresponding S flow The convolution weights of the convolutional layers, where all convolutional layers used can be 1*1*1 convolutional layers, W1 represents the weights of the fully connected layers, β represents batch normalization, and δ represents the activation function.

[0089] In another possible implementation, determining the weights of the multimodal information can also be carried out as follows:

[0090] Step E1: Use a self-attention module to extract the self-attention features of various information in the multimodal information.

[0091] In the self-attention mechanism, K (key), V (value), and Q (query) are all information of the corresponding modality.

[0092] Step E2: For the self-attention features of each pair of information in the multimodal information, the cross-attention module is used to extract cross-attention features, resulting in multiple cross-attention features.

[0093] For any two types of information with self-attention features, one self-attention feature is designated as the first feature, and the other as the second feature. The first feature serves as Q of the cross-attention module, and the second feature serves as K and V of the cross-attention module. For example, if the self-attention feature of color image information is S1, the self-attention feature of depth information is S2, and the self-attention feature of optical flow information is S3, then each group (S1, S2), (S1, S3), and (S2, S3) is processed based on the cross-attention module. The cross-attention feature obtained from (S1, S2) is V1, the cross-attention feature obtained from (S1, S3) is V2, and the cross-attention feature obtained from (S2, S3) is V3.

[0094] Step E3 involves using a fully connected layer to process multiple cross-attention features and obtain the weights of each modality.

[0095] In summary, each type of information is extracted using an attention module, and then the weights obtained are learned from each other using a cross-attention module. By referencing information from other modalities, reasonable weights can be learned by comprehensively considering the situation of other modalities.

[0096] In this embodiment, the weights of each modality are determined based on information from various modalities, and then the information from various modalities is fused, so that the fused structure retains as much key information as possible from each modality. This avoids the situation where complex processing is performed on the information from various modalities before fusion, as this complex processing eliminates some key features. Therefore, the method of fusing first and then extracting gesture features can improve the accuracy of dynamic gesture recognition.

[0097] In summary, the recognition model used for recognizing dynamic gestures in the embodiments of this disclosure is as follows: Figure 3 As shown, it may include: an adaptive fusion network, a feature extraction network, a temporal convolutional network, and a classification network. The adaptive fusion network is used to fuse multimodal information to obtain fused information; the fused information is input to the feature extraction network to obtain short-term spatiotemporal features of the dynamic gesture; the short-term spatiotemporal features are input to the temporal convolutional network to obtain long-term spatiotemporal features; and the classification network classifies the long-term spatiotemporal features to obtain the probability distribution of the dynamic gesture.

[0098] The entire training process of the recognition model can be summarized as follows: Figure 4 As shown:

[0099] S401 preprocesses a large number of sample videos to obtain multimodal information, including color image information, depth information, and optical flow information.

[0100] S402 uses a large number of sample videos to pre-train the feature extraction network.

[0101] The feature extraction network can be a 3D residual network or a T3D network.

[0102] During pre-training, multimodal information can be input into the feature extraction network to complete its pre-training. Alternatively, multimodal information can be fused and then input into the feature extraction network for pre-training. Furthermore, partial information from the multimodal data, such as color image information and depth information, can also be input into the feature extraction network for pre-training. In implementation, the pre-training method can be determined based on actual needs.

[0103] After pre-training, a lightweight feature extraction network weight file is saved in .h5 format. The weight file saves the model structure, model weights, training configuration, and optimizer state, so that formal training can resume from where it was interrupted last time.

[0104] S403 uses sample videos to formally train the recognition model. During formal training, the training and validation sets of multimodal information are used as inputs, and an adaptive fusion network is used for fusion.

[0105] like Figure 4 During the formal training process, steps S4031 involve acquiring multimodal information from sample videos. In S4032, this multimodal information is fused using an adaptive fusion network to obtain fused information. In S4033, the fused information is input into a pre-trained feature extraction network to obtain short-term spatiotemporal features. In S4034, these short-term spatiotemporal features are input into a temporal convolutional network to obtain long-term spatiotemporal features. In S4035, the long-term spatiotemporal features are then input into a classification network to obtain the classification result for the dynamic gestures. In S4036, this classification result and the classification label are used to determine the loss, and the parameters of the recognition model are adjusted based on the loss. This process is repeated until the convergence condition is met.

[0106] Once the recognition model is trained, the overall dynamic gesture recognition process can also be represented as follows: Figure 5 As shown, after extracting the RGB sequence (i.e., color image information), Depth sequence (i.e., depth information), and optical flow (i.e., optical flow information) of dynamic gestures, they are weighted and summed using an adaptive fusion network to obtain fused information. This fused information is then input into the T3D network to extract short-term spatiotemporal features, and then into the TCN network to obtain long-term spatiotemporal features. Finally, it is input into the fully connected layer and softMax (classification layer) of the classification network to obtain the recognition result of the dynamic gestures.

[0107] Based on the same technical concept, this disclosure also provides a gesture recognition device 600, such as... Figure 6 As shown, it includes:

[0108] Information acquisition module 601 is used to acquire multimodal information of the dynamic gesture to be recognized based on the video of the dynamic gesture to be recognized;

[0109] The fusion module 602 is used to fuse multimodal information to obtain fused information of the dynamic gesture to be recognized;

[0110] The feature extraction module 603 is used to determine the gesture features of the dynamic gesture to be recognized based on the fused information;

[0111] The recognition module 604 is used to determine the gesture category of the dynamic gesture to be recognized based on gesture features.

[0112] In some embodiments, the multimodal information includes at least one of the following: color image information, depth information, and optical flow information.

[0113] In some embodiments, the information acquisition module includes:

[0114] The preprocessing unit is used to process the video frame number to the target frame number through a sampling or interpolation device to obtain the image sequence of the dynamic gesture to be recognized;

[0115] The extraction unit is used to extract multimodal information from the image sequence.

[0116] In some embodiments, where the multimodal information includes optical flow information, the extraction unit is configured to:

[0117] Divide the image sequence into n parts to obtain n subsequences, where n is a positive integer greater than 1;

[0118] For each subsequence, perform the following: determine the importance of the subsequence and use the importance as the weight of the optical flow graph of the subsequence;

[0119] The optical flow maps of each subsequence in the original optical flow map of the image sequence are weighted and summed to obtain the optical flow information of the dynamic gesture to be recognized.

[0120] In some embodiments, the extraction unit is specifically used for:

[0121] Determine the mean optical flow of the optical flow map of the subsequence;

[0122] The ratio between the mean optical flow and the mean optical flow of the image sequence is used to determine the importance of the subsequence.

[0123] In some embodiments, the extraction unit is specifically used for:

[0124] Determine at least one of the following importance metrics: image quality of the subsequence, gesture integrity, and the degree of data difference within the multi-frame optical flow maps of the subsequence;

[0125] The importance of subsequences is determined based on the positive correlation between each importance indicator and the degree of importance.

[0126] In some embodiments, the fusion module is configured to:

[0127] The self-attention mechanism is used to perform nonlinear fusion of multimodal information to obtain the fused information of the dynamic gesture to be recognized.

[0128] In some embodiments, the fusion module includes:

[0129] The weight determination unit is used to determine the weights of each piece of information in multimodal information based on the self-attention mechanism;

[0130] The fusion unit is used to perform weighted summation of multimodal information based on the weights of each piece of information to obtain the fused information of the dynamic gesture to be recognized.

[0131] The specific functions and examples of each module and unit of the apparatus in this disclosure embodiment can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0132] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0133] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0134] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0135] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0136] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as gesture recognition methods. For example, in some embodiments, the gesture recognition method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the gesture recognition method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform gesture recognition methods by any other suitable means (e.g., by means of firmware).

[0137] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0138] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0139] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0140] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0141] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0142] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0143] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0144] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A gesture recognition method, comprising: Based on a video of a dynamic gesture to be recognized, multimodal information of the gesture is obtained, including optical flow information. Obtaining the optical flow information includes: determining an image sequence based on the video; dividing the image sequence into n parts to obtain n sub-sequences, where n is a positive integer greater than 1; determining the importance of each sub-sequence; and using the importance of each sub-sequence as a weight, performing a weighted summation of the optical flow maps of each sub-sequence in the original optical flow map of the image sequence to obtain the optical flow information. Determining the importance of each sub-sequence includes: for each sub-sequence, performing the following steps: determining the mean optical flow of the sub-sequence's optical flow map, and determining the ratio between the mean optical flow and the mean optical flow of the image sequence as the importance of the sub-sequence. The multimodal information is fused to obtain the fused information of the dynamic gesture to be recognized; Based on the fused information, the gesture features of the dynamic gesture to be identified are determined; The gesture category of the dynamic gesture to be recognized is determined based on the gesture features.

2. The method according to claim 1, wherein, The multimodal information also includes at least one of the following: color image information and depth information.

3. The method according to claim 1 or 2, wherein, Based on the video of the dynamic gesture to be recognized, obtain the multimodal information of the dynamic gesture to be recognized, including: The number of frames in the video is processed to the target number of frames by sampling or interpolation methods to obtain the image sequence of the dynamic gesture to be recognized; The multimodal information is extracted from the image sequence.

4. The method according to claim 1, further comprising determining the importance of the subsequence: Determine at least one of the following importance metrics: image quality of the subsequence, gesture integrity, and the degree of data difference within the multi-frame optical flow graph of the subsequence; The importance of the subsequence is determined based on the positive correlation between each importance indicator and the importance level.

5. The method according to claim 1, wherein, The multimodal information is fused to obtain the fused information of the dynamic gesture to be recognized, including: The multimodal information is nonlinearly fused based on a self-attention mechanism to obtain the fused information of the dynamic gesture to be recognized.

6. The method according to claim 5, wherein, The multimodal information is nonlinearly fused based on a self-attention mechanism to obtain the fused information of the dynamic gesture to be recognized, including: The weights of each piece of information in the multimodal information are determined based on a self-attention mechanism; Based on the weights of each piece of information, the multimodal information is weighted and summed to obtain the fused information of the dynamic gesture to be recognized.

7. A gesture recognition device, comprising: An information acquisition module is used to acquire multimodal information of a dynamic gesture to be recognized based on a video of the gesture. The multimodal information includes optical flow information. Acquiring the optical flow information includes: determining an image sequence based on the video; dividing the image sequence into n parts to obtain n sub-sequences, where n is a positive integer greater than 1; determining the importance of each sub-sequence; and using the importance of each sub-sequence as a weight, performing a weighted summation of the optical flow maps of each sub-sequence in the original optical flow map of the image sequence to obtain the optical flow information. Determining the importance of each sub-sequence includes: for each sub-sequence, performing the following steps: determining the mean optical flow of the optical flow map of the sub-sequence, and determining the ratio between the mean optical flow and the mean optical flow of the image sequence as the importance of the sub-sequence. The fusion module is used to fuse the multimodal information to obtain the fused information of the dynamic gesture to be recognized; The feature extraction module is used to determine the gesture features of the dynamic gesture to be identified based on the fused information; The recognition module is used to determine the gesture category of the dynamic gesture to be recognized based on the gesture features.

8. The apparatus according to claim 7, wherein, The multimodal information also includes at least one of the following: color image information and depth information.

9. The apparatus according to claim 7 or 8, wherein, The information acquisition module includes: The preprocessing unit is used to process the number of frames in the video to a target number of frames using a sampling or interpolation device, thereby obtaining the image sequence of the dynamic gesture to be recognized. An extraction unit is used to extract the multimodal information from the image sequence.

10. The apparatus according to claim 9, wherein the extraction unit is specifically used for: Determine at least one of the following importance metrics: image quality of the subsequence, gesture integrity, and the degree of data difference within the multi-frame optical flow graph of the subsequence; The importance of the subsequence is determined based on the positive correlation between each importance indicator and the importance level.

11. The apparatus according to claim 7, wherein, The fusion module is used for: The multimodal information is nonlinearly fused based on a self-attention mechanism to obtain the fused information of the dynamic gesture to be recognized.

12. The apparatus according to claim 11, wherein, The fusion module includes: The weight determination unit is used to determine the weight of each piece of information in the multimodal information based on the self-attention mechanism; The fusion unit is used to perform weighted summation of the multimodal information based on the weights of each piece of information to obtain the fusion information of the dynamic gesture to be recognized.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images