Gesture recognition method and device and electronic equipment

Abstract

The invention provides a gesture recognition method and device and electronic equipment, and relates to the technical field of recognition, and the gesture recognition method comprises the steps: obtaining a video frame sequence, traversing video frames in the video frame sequence, and inputting the video frames into a gesture recognition network to obtain a gesture detection result, including gesture coordinates, gesture categories, original confidence and similarity; generating a confidence coefficient queue based on the original confidence coefficient, and generating a standard confidence coefficient of the gesture detection result according to the confidence coefficient queue; determining whether the video frame is a target video frame according to the standard confidence and the similarity, and calculating a first gesture parameter according to a sequence value corresponding to the target video frame when the video frame is the target video frame; calculating a second gesture parameter according to the standard confidence and the similarity of the target video frame; and generating a gesture recognition result based on the standard confidence, the first gesture parameter and the second gesture parameter. According to the method, the accuracy of gesture recognition can be improved, and the situation of gesture false detection caused by gesture transformation is reduced.

Description

Technical Field

[0001] The present application relates to the field of recognition technology, and in particular to a gesture recognition method, device and electronic device. Background Art

[0002] Gesture recognition is a human-computer interaction technology that uses computer vision, sensors and other technical means to capture, analyze and understand the movements and postures of human hands, thereby achieving interaction between users and devices. Gesture recognition can be applied to technical fields such as smart homes, entertainment games, medical assistance and vehicle control.

[0003] By collecting hand images through optical sensing devices such as cameras, and extracting gesture-related features from the collected images based on image processing and deep learning, and matching the extracted features with predefined gesture models, the meaning of the current gesture can be determined.

[0004] However, since the above method uses the hand key point data as the basic data when extracting gesture-related features, it is easy to fail to obtain clear hand key points when the acquisition distance is far or the acquisition scene is blurred, which leads to inaccurate gesture recognition. In addition, since the gesture will change when the optical sensing device collects the hand image, similar gestures will appear in the process of gesture change, causing the extracted gesture features to mismatch with the predefined gesture model, resulting in a decrease in the accuracy of gesture recognition. Summary of the invention

[0005] In order to solve the above problems, the present application provides a gesture recognition method, device and electronic device, which can reduce the probability of gesture misdetection during gesture change, thereby improving the accuracy of gesture recognition.

[0006] In a first aspect, the present application provides a gesture recognition method, comprising: obtaining a video frame sequence, the video frame sequence comprising video frames and sequence values ​​corresponding to the video frames, the sequence value being a positive integer greater than or equal to 1; Traversing the video frames in the video frame sequence, and inputting the video frames into a gesture recognition network to obtain gesture detection results corresponding to the video frames, wherein the gesture recognition network is a deep neural network constructed based on a similarity score, and the gesture detection results include gesture coordinates, gesture categories, original confidence, and similarity; Generate a confidence queue based on the original confidence, and generate a standard confidence of the gesture detection result according to the confidence queue; Determine the video frame whose standard confidence is greater than or equal to a preset confidence threshold and whose similarity is greater than or equal to a preset similarity threshold as a target video frame; and determine the video frame whose standard confidence is less than the preset confidence threshold or whose similarity is less than the preset similarity threshold as a non-target video frame; In the case where the video frame is the target video frame, a first gesture parameter is calculated according to a sequence value corresponding to the target video frame; and a second gesture parameter is calculated according to the standard confidence and the similarity of the target video frame; A gesture recognition result is generated based on the standard confidence, the first gesture parameter, and the second gesture parameter, where the gesture recognition result is used to characterize the gesture category and gesture position of the target video frame.

[0007] In an optional implementation, before inputting the video frame into a gesture recognition network to obtain a gesture detection result corresponding to the video frame, the method further includes: obtaining a case data set, the case data set including at least one training gesture data, the training gesture data being provided with a gesture annotation area and a gesture annotation category; Generate a standard gesture label according to the training gesture data, and calculate a similarity score between the training gesture data and the standard gesture label, wherein the similarity score is used to characterize the difference between the training gesture data and the standard gesture label, and the similarity score is associated with the training gesture data; Inputting the training gesture data and the similarity score into the gesture recognition network, and controlling the gesture recognition network to output regression parameters, wherein the regression parameters include coordinate information, size information, confidence information, classification result information, and similarity information; The gesture recognition network is trained based on the regression parameters, the gesture annotation categories and the loss function to optimize the model parameters of the gesture recognition network.

[0008] In an optional embodiment, generating a standard gesture label according to the training gesture data includes: grouping the training gesture data according to the gesture annotation category to obtain at least one group of gesture classification groups; Acquire a target gesture area of ​​the training gesture data according to the gesture annotation area, and input the target gesture area into a feature extraction model to obtain gesture features, wherein the gesture features are associated with the training gesture data; A standard gesture label is generated according to the gesture feature, and the standard gesture label is used to characterize a reference gesture of the standard gesture category.

[0009] In an optional implementation, the training gesture data includes continuous frame data and non-continuous frame data; when the training gesture data is the continuous frame data, the gesture features of the training gesture data are marked as continuous frame gesture features; In the case where the training gesture data is the non-continuous frame data, a preset weight is multiplied by the gesture feature of the training gesture data to obtain a non-continuous frame gesture feature.

[0010] In an optional embodiment, generating a standard gesture label according to the gesture features includes: obtaining reference gesture features of the training gesture data in each of the gesture category groups, the reference gesture features including the continuous frame gesture features and / or the non-continuous frame gesture features; The average value of the reference gesture features is calculated to obtain the standard gesture label.

[0011] In an optional implementation, calculating the similarity score between the training gesture data and the standard gesture label includes: acquiring reference gesture data, the reference gesture data being a standard gesture label of a gesture classification group to which the training gesture data belongs; A cosine distance between the training gesture data and the reference gesture data is calculated to obtain the similarity score.

[0012] In an optional implementation, the loss function includes class loss, target loss, position loss and similarity loss, the similarity loss is the quotient of the category edit distance and the number of categories, the category edit distance is used to characterize the difference between the classification result information and the gesture annotation category, and the number of categories is used to characterize the number of gesture classification groups; The step of training the gesture recognition network based on the regression parameter, the gesture annotation category and the loss function to optimize the model parameters of the gesture recognition network includes: Optimizing classification parameters of the gesture recognition network based on the class loss, wherein the classification parameters are used to calculate the classification result information; Optimizing a confidence parameter of the gesture recognition network based on the target loss, wherein the confidence parameter is used to calculate the confidence information; Optimizing position parameters of the gesture recognition network based on the position loss, wherein the position parameters are used to calculate the coordinate information, width information, and height information; A similarity parameter of the gesture recognition network is optimized based on the similarity loss, and the similarity parameter is used to calculate the similarity information.

[0013] In an optional implementation, generating a confidence queue based on the original confidence, and generating a standard confidence of the gesture detection result according to the confidence queue, including: when the sequence value of the video frame is equal to 1, the standard confidence of the gesture detection result is the original confidence; When the sequence value of the video frame is greater than 1, a reference confidence is obtained, where the reference confidence is the original confidence of the reference frame, and the sequence value corresponding to the reference frame is less than the sequence value corresponding to the video frame; and the confidence queue is generated according to the reference confidence and the original confidence, and the average value of the reference confidence and the original confidence is calculated to obtain the standard confidence of the gesture detection result.

[0014] In an optional embodiment, the gesture recognition method further includes: when the video frame is the non-target video frame, inputting a backward video frame into the gesture recognition network to perform gesture recognition on the backward video frame, the backward video frame being the next frame of the non-target video frame.

[0015] In an optional implementation, calculating the first gesture parameter according to the sequence value corresponding to the target video frame includes: generating a state index according to the sequence value corresponding to the target video frame, the state index being used to represent the number of target video frames in the video frame sequence from the video frame with the sequence value of 1 to the current video frame; The first gesture parameter is calculated using a weighted average formula, where the weighted average formula is: , in, represents the state index, represents the first gesture parameter, represents the intensity factor, which is calculated using the intensity formula, which is: , in, Indicates the preset frame number. Indicates the preset step size.

[0016] In an optional implementation, calculating the second gesture parameter according to the standard confidence and the similarity includes: when the sequence value of the target video frame is equal to 1, setting the second gesture parameter to a first preset value; When the sequence value of the target video frame is greater than 1, a forward confidence and a forward similarity are obtained, wherein the forward confidence is a standard confidence of the forward video frame, the forward similarity is a similarity of the forward video frame, and the forward video frame is a previous frame of the target video frame; When the forward confidence is less than the standard confidence of the target video frame, and the forward similarity is less than the similarity of the target video frame, setting the second gesture parameter to a first preset value; When the forward confidence is greater than or equal to the standard confidence of the target video frame, and the forward similarity is greater than or equal to the similarity of the target video frame, the second gesture parameter is set to a second preset value.

[0017] In an optional implementation, generating a gesture recognition result based on the standard confidence, the first gesture parameter, and the second gesture parameter includes: multiplying the standard confidence, the first gesture parameter, and the second gesture parameter to obtain a first detection result; Performing non-maximum suppression algorithm processing on the first detection result to obtain a second detection result; The second detection result is decoded to generate the gesture recognition result.

[0018] In a second aspect, the present application provides a gesture recognition device, including: a video frame acquisition module, configured to: obtain a video frame sequence, the video frame sequence including video frames and sequence values ​​corresponding to the video frames, the sequence value being a positive integer greater than or equal to 1; A gesture detection module is configured to: traverse the video frames in the video frame sequence, input the video frames into a gesture recognition network to obtain a gesture detection result corresponding to the video frame, wherein the gesture recognition network is a deep neural network constructed based on a similarity score, and the gesture detection result includes gesture coordinates, gesture category, original confidence and similarity; A confidence processing module is configured to: generate a confidence queue based on the original confidence, and generate a standard confidence of the gesture detection result according to the confidence queue; The video frame determination module is configured to: determine the video frame whose standard confidence is greater than or equal to a preset confidence threshold and whose similarity is greater than or equal to a preset similarity threshold as a target video frame; and determine the video frame whose standard confidence is less than the preset confidence threshold or whose similarity is less than the preset similarity threshold as a non-target video frame; The result output module is configured to: when the video frame is the target video frame, calculate the first gesture parameter according to the sequence value corresponding to the target video frame; and calculate the second gesture parameter according to the standard confidence and the similarity of the target video frame; generate a gesture recognition result based on the standard confidence, the first gesture parameter and the second gesture parameter, and the gesture recognition result is used to characterize the gesture category and gesture position of the target video frame; In a case where the video frame is the non-target video frame, a backward video frame is input into the gesture recognition network to perform gesture recognition on the backward video frame, where the backward video frame is a next frame of the non-target video frame.

[0019] In a third aspect, the present application provides an electronic device comprising: a memory and one or more processors; the memory is configured to: store one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the gesture recognition method of the first aspect.

[0020] As can be seen from the above technical solutions, the present application provides a gesture recognition method, device and electronic device. The gesture recognition method includes: obtaining a video frame sequence, traversing the video frames in the video frame sequence, and inputting the video frames into a gesture recognition network to obtain gesture detection results, including gesture coordinates, gesture categories, original confidence and similarity; generating a confidence queue based on the original confidence, and generating a standard confidence of the gesture detection result based on the confidence queue; determining whether the video frame is a target video frame based on the standard confidence and similarity, and if the video frame is a target video frame, calculating a first gesture parameter based on the sequence value corresponding to the target video frame; and calculating a second gesture parameter based on the standard confidence and similarity of the target video frame; generating a gesture recognition result based on the standard confidence, the first gesture parameter and the second gesture parameter. The above method can improve the accuracy of gesture recognition and reduce the situation of gesture misdetection due to gesture change. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 A flowchart of a gesture recognition method provided in this embodiment; Figure 2 A schematic diagram of a training process of a gesture recognition network provided in this embodiment; Figure 3 A schematic diagram of regression parameters of a gesture recognition network provided in this embodiment; Figure 4 A schematic diagram of a method for generating standard confidence provided in this embodiment; Figure 5 A schematic diagram of the structure of a gesture recognition device provided in this embodiment; Figure 6 A schematic diagram of an electronic device provided in this embodiment. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0024] It should be understood that the "multiple" mentioned herein refers to two or more. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not limit them to be necessarily different.

[0025] In addition, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, method, product, or apparatus.

[0026] Gesture recognition is a human-computer interaction technology that uses computer vision, sensors and other technical means to capture, analyze and understand the movements and postures of human hands, thereby achieving interaction between users and devices. Gesture recognition can be applied to technical fields such as smart homes, entertainment games, medical assistance and vehicle control.

[0027] In some embodiments, the user wears a wearable device with sensors such as gloves or wristbands and makes gestures. The wearable device captures the changes in electrical signals, converts the electrical signals into digital signals, and then analyzes and recognizes the gestures. This method can overcome the influence of the external environment and accurately capture hand movements and postures, but it needs to rely on the device support of the wearable device.

[0028] In other embodiments, optical sensing devices such as cameras are used to collect images of the user's hands, and gestures are distinguished through image processing and deep learning. This method does not require the support of external devices, can improve the user experience, and is suitable for the recognition of user gestures in scenarios with high real-time and freedom requirements.

[0029] However, when using optical sensing devices such as cameras to collect user hand images, it is easy to fail to obtain clear hand key points when the collection distance is far or the collection scene is blurred, which leads to inaccurate gesture recognition. In addition, since the gesture will change when the optical sensing device collects the hand image, similar gestures will appear during the gesture change process, causing the extracted gesture features to mismatch with the predefined gesture model, resulting in a decrease in the accuracy of gesture recognition.

[0030] In order to reduce the probability of false detection of gestures during gesture transformation and improve the accuracy of gesture recognition, the first aspect of the present application provides a gesture recognition method. Figure 1 A flow chart of a gesture recognition method provided in this embodiment is shown below. Figure 1 The gesture recognition method of the embodiment of the present application is described in detail.

[0031] Step S100: Obtain a video frame sequence.

[0032] In the embodiment of the present application, a video containing gestures can be obtained by a camera device. It should be understood that a video is essentially a visual effect formed by a series of continuous images played quickly, and each individual image in these continuous images is called a video frame. The video is processed by video processing software to generate a video frame sequence. Among them, the video frame sequence includes video frames and sequence values ​​corresponding to the video frames, and the sequence value is a positive integer greater than or equal to 1.

[0033] The sequence value is used to represent the order of the video frames in the video frame sequence. For example, the sequence value of the first video frame is 1, the sequence value of the second video frame is 2, and the sequence value of the nth video frame is n.

[0034] In some embodiments, a video containing gestures may be processed frame by frame using a sliding window to obtain video frames. The sliding window maintains a window of a specific size, slides in the video at a set step length, and performs image processing, such as object detection, feature extraction, and image classification, on the image in the window to obtain a video frame.

[0035] Step S200: traverse the video frames in the video frame sequence, and input the video frames into a gesture recognition network to obtain gesture detection results corresponding to the video frames.

[0036] In an embodiment of the present application, the gesture recognition network is a deep neural network constructed based on similarity scores, specifically, a neural network model used to detect specific targets or objects in the field of computer vision and artificial intelligence. After the video frame is input into the gesture recognition network, the gesture recognition network outputs the gesture detection result corresponding to the video frame after feature extraction, target classification and post-processing. Among them, the gesture detection result includes gesture coordinates, gesture category, original confidence and similarity.

[0037] In the process of processing video frames by the gesture recognition network, the gesture recognition network first uses a feature extractor constructed by convolutional layers, pooling layers, etc. to process the input image or video frame to convert the original image data into feature maps, which contain various semantic information and spatial information in the image. After obtaining the feature map, the detection network predicts the location and category of the target through a series of prediction heads or branches. For the prediction of the target location, the bounding box coordinates of the target are output, such as the coordinates of the upper left corner and the lower right corner, or the center coordinates and width and height information. For the prediction of the target category, the probability or score of each possible category is output to indicate the possibility of the existence of a certain category of target at the current location. Finally, the results obtained by network prediction usually need to be post-processed to obtain the final detection result. Common post-processing operations include non-maximum suppression (NMS), which is used to remove redundant detection boxes with high overlap and retain the detection box with the highest score as the final result.

[0038] In some embodiments, in order to improve the accuracy of gesture detection results, it is necessary to build a trainable gesture recognition network based on the similarity score. Figure 2 As shown, the step of constructing a gesture recognition network includes steps S210 to S240.

[0039] Step S210: Obtain case data set.

[0040] In some embodiments, the electronic device obtains the case data set by means of a web crawler, querying a professional data set, etc. The case data set includes at least one training gesture data, and the training gesture data is used to characterize different gesture videos. For example, the training gesture data may be different gesture types, different shooting angles, different lighting conditions, and different shooting backgrounds.

[0041] The electronic device annotates the collected training gesture data to mark the category to which each gesture belongs and the area where the gesture is located. The training gesture data is preprocessed, such as normalization, cropping, scaling, data enhancement, etc. Among them, normalization can scale the pixel values ​​of the training gesture data to a preset range, cropping and scaling can adjust the image to a uniform size, and data enhancement can expand the data set by rotating, flipping, and adding noise, thereby increasing the diversity of the training gesture data.

[0042] Each training gesture data is provided with a gesture annotation area and a gesture annotation category. The gesture annotation area is used to characterize the coordinates of the gesture in the training gesture data, and the gesture annotation category is used to characterize the type of gesture in the training gesture data, such as: indicating gesture, thumbs-up gesture, and number gesture.

[0043] In this way, the gesture recognition network can train the gesture annotation area and gesture annotation category of the gesture data in the case data set, identify the shape, texture and other characteristics of different gestures, and improve the accuracy of gesture recognition.

[0044] Step S220: Generate a standard gesture label according to the training gesture data, and calculate a similarity score between the training gesture data and the standard gesture label.

[0045] In some embodiments, the electronic device groups the training gesture data in the case data set according to the gesture annotation category set in the training gesture data to obtain at least one group of gesture classification groups, such as an indication gesture group, a thumbs-up gesture group, and a quantity gesture group.

[0046] Next, the electronic device obtains the target gesture area of ​​the training gesture data according to the gesture annotation area, and inputs the target gesture area into the feature extraction model to obtain gesture features. Among them, the gesture features are associated with the training gesture data, that is, each training gesture data has its corresponding gesture features. Finally, a standard gesture label is generated according to the gesture features, and the standard gesture label is used to characterize the reference gesture of the gesture annotation category.

[0047] It should be noted that the feature extraction model is a convolutional neural network. The convolution layer of the feature extraction model performs convolution operations by sliding the convolution kernel on the image, automatically extracting local features of the gesture image, such as low-level features such as edges, corners, and textures. As the number of network layers increases, the high-level convolution layers can combine these low-level features into more abstract and representative high-level features, such as the combination pattern of fingers, the shape structure of the overall gesture, etc. The pooling layer can compress and downsample the features extracted by the convolution layer, reducing the amount of data while retaining the main features, and improving the computational efficiency and generalization ability of the model.

[0048] In some embodiments, the training gesture data includes continuous frame data and discontinuous frame data, wherein the continuous frame data is used to represent the gesture action and is stored in a continuous video frame, and the discontinuous frame data is used to represent the gesture action and is stored in multiple video frames, and the multiple video frames constitute the discontinuous frame data.

[0049] When the training gesture data is continuous frame data, the gesture features of the training gesture data are marked as continuous frame gesture features; when the training gesture data is discontinuous frame data, the preset weight is multiplied by the gesture features of the training gesture data to obtain discontinuous frame gesture features.

[0050] In some embodiments, a method for generating a standard gesture label based on gesture features is: obtaining reference gesture features of the training gesture data in each gesture category grouping, the reference gesture features including continuous frame gesture features and / or non-continuous frame gesture features; and calculating the average value of the reference gesture features to obtain a standard gesture label.

[0051] Exemplarily, the first gesture group includes first continuous frame data and first non-continuous frame data, and the first continuous frame gesture features and the first non-continuous frame gesture features are obtained, wherein the first non-continuous frame gesture features are features processed by exponential moving average (EMA). The first continuous frame gesture features and the first non-continuous frame gesture features are averaged to generate a standard gesture label for the first gesture group. In the above process, the exponential moving average is a weighted moving average, which processes the data according to preset weights to smooth the data and reduce short-term noise and fluctuations in the data.

[0052] In this way, by calculating the average value of the reference gesture features and obtaining the standard gesture label, the feature result can be unaffected by a single outlier, thereby improving the accuracy of the standard gesture label.

[0053] Furthermore, the electronic device calculates a similarity score between the training gesture data and the standard gesture label, wherein the similarity score is used to characterize the difference between the training gesture data and the standard gesture label, and the similarity score is associated with the training gesture data.

[0054] In some embodiments, the similarity score is calculated using cosine similarity, including: obtaining reference gesture data, where the reference gesture data is a standard gesture label of the gesture classification group where the training gesture data is located; and calculating the cosine distance between the training gesture data and the reference gesture data, thereby obtaining a similarity score.

[0055] It should be noted that the method for calculating the similarity score also includes: using the Euclidean distance to calculate the distance between the training gesture data and the reference gesture data, wherein the smaller the Euclidean distance, the more similar the training gesture data and the reference gesture data are. This application does not specifically limit the method for calculating the similarity score.

[0056] By calculating the similarity between the training gesture data and the reference gesture data, the gesture recognition network can be provided with a data basis for gesture recognition, thereby improving the reliability of the gesture recognition network for gesture recognition.

[0057] Step S230: input the training gesture data and the similarity score into the gesture recognition network, and control the gesture recognition network to output regression parameters.

[0058] The regression parameters include coordinate information, size information, confidence information, classification result information and similarity information.

[0059] In some embodiments, in order to ensure the real-time performance of gesture recognition, the gesture recognition network uses YOLO (YouOnly Look Once) as the basic network architecture. YOLO is a real-time target detection algorithm widely used in the field of computer vision. It uses a single-stage test framework to unify the target detection task into a regression problem, and directly predicts the position and category of the target in a forward propagation process. It has a fast detection speed and can meet application scenarios with high real-time requirements.

[0060] The structure of YOLO includes the backbone network (Backbone), the neck network (Neck) and the head network (Head). Among them, the backbone network is used to extract the features of the image. The backbone network usually contains multiple convolutional layers and pooling layers. Through continuous convolution operations, it extracts different levels of image features, such as low-level features such as edges and textures to more abstract semantic features.

[0061] The neck network is located between the backbone network and the detection head and is used for feature fusion and processing. For example, the spatial pyramid pooling layer can fuse features of different scales, and the feature pyramid network can transfer and fuse information between feature maps at different levels, thereby enhancing the model's ability to detect objects of different scales.

[0062] The head network is used to output the detection results, including the category and location information of the target. It usually contains convolutional layers and regression layers, and predicts the bounding box coordinates, category probability, etc. of the target through convolution operations on feature maps.

[0063] In order to improve the scalability and flexibility of the gesture recognition network, the embodiment of the present application adopts YOLOV5n as the basic network architecture so that the gesture recognition network can adjust the network depth and width as needed, and support data enhancement technology and training strategies, thereby improving the performance and robustness of the gesture recognition network.

[0064] Furthermore, the electronic device inputs the training gesture data and the similarity score into the gesture recognition network, and controls the gesture recognition network to output regression parameters, where the regression parameters include coordinate information, size information, confidence information, classification result information, and similarity information.

[0065] In some embodiments, the gesture recognition network applies a convolutional layer to extract features from the input gesture image and outputs three branches, each of which has parameters including coordinate information, size information, confidence information conf, and classification result information class. Figure 3 As shown, the coordinate information includes the horizontal coordinate x and the vertical coordinate y, and the size information includes the width w and the height h. Next, the parameters output by each branch are adjusted, and the similarity information m is added to generate regression parameters. The regression parameters include coordinate information, size information, confidence information conf, classification result information class, and similarity information m. Similarly, the coordinate information includes the horizontal coordinate x and the vertical coordinate y, and the size information includes the width w and the height h. In this way, by adding the similarity information m, the accuracy of gesture recognition can be constrained, thereby improving the accuracy of the output results of the gesture recognition network.

[0066] Step S240: training the gesture recognition network based on the regression parameters, the gesture standard categories and the loss function to optimize the model parameters of the gesture recognition network.

[0067] The loss function is a function used in machine learning and deep learning to measure the difference between the model's predicted results and the actual results. The loss function can convert the difference between the predicted results (i.e., regression parameters) output by the gesture recognition network and the actual results (i.e., gesture standard categories) into a numerical value, which can be used as a feedback signal for the training of the gesture recognition network. The gesture recognition network continuously optimizes the model parameters through the optimization algorithm to reduce the value of the loss function.

[0068] In some embodiments, the loss function includes class loss, target loss, position loss and similarity loss. The gesture recognition network is trained based on the regression parameter, gesture annotation category and loss function to optimize the model parameters of the gesture recognition network, including: optimizing the classification parameters of the gesture recognition network based on the class loss, the classification parameters are used to calculate the classification result information; optimizing the confidence parameters of the gesture recognition network based on the target loss, the confidence parameters are used to calculate the confidence information; optimizing the position parameters of the gesture recognition network based on the position loss, the position parameters are used to calculate the coordinate information, width information and height information; optimizing the similarity parameters of the gesture recognition network based on the similarity loss, the similarity parameters are used to calculate the similarity information.

[0069] Exemplarily, the loss function can be expressed using the following formula: ; (1) in, represents the loss function, represents the class loss, represents the weight of class loss, represents the target loss, represents the weight of the target loss, represents the position loss, represents the weight of the position loss, represents the similarity loss, Represents the weight of the similarity loss.

[0070] In some embodiments, the gesture recognition network can calculate the intersection-and-union ratio of the predicted gesture area and the real gesture area based on the target loss, and then evaluate the accuracy and effect of the gesture recognition network to optimize the confidence parameter. Among them, the predicted gesture area is the predicted area output by the gesture recognition network, and the real gesture area is a pre-set anchor box, which is used to characterize the width and height of the common gesture area. For example, the intersection-and-union ratio of the predicted gesture area and the real gesture area can be calculated by dividing the overlapping area of ​​the predicted gesture area and the real gesture area by the combined area of ​​the predicted gesture area and the real gesture area.

[0071] In some embodiments, the similarity loss is the quotient of the class edit distance and the number of classes, wherein the class edit distance is used to characterize the difference between the classification result information and the gesture annotation class, and the number of classes is used to characterize the number of gesture classification groups. The similarity loss can be calculated using the following formula: ; (2) in, represents the similarity loss, Indicates the standard category of gestures, Indicates the classification result information. Indicates the number of categories, Represents the category edit distance.

[0072] A loss function is generated based on the category edit distance, and a gesture recognition network is trained based on the loss function, so that the gesture recognition network can process the input video frame sequence in real time. If there is a gesture target in the video frame, the gesture recognition network outputs the coordinate information, width information, size information, confidence information, classification result information and similarity information of the gesture.

[0073] Exemplarily, in the process of calculating the category edit distance between the gesture standard category and the classification result information, the gesture standard category is initialized to the first string a, the classification result information is initialized to the second string b, and the Represents the category edit distance between the first m characters of the first string a and the first n characters of the second string b. Set , used to characterize the category edit distance from the empty string to the second string b of length n as n insertion operations, and set , used to characterize that the category edit distance from the first string a of length m to the empty string is m deletion operations.

[0074] When m>0 and n>0, the category edit distance is calculated by determining whether the characters of the first string a and the characters of the second string b are equal, including: if (that is, the m-1th character in the first string a is the same as the n-1th character in the second string b), then ;like Not equal to (i.e. the first The first character in the second string b characters are different), then the smallest result value among the deletion, insertion, and replacement operations is obtained as the category edit distance. The result value of the deletion operation is The result value of the delete operation is used to represent the deletion of the mth character of the first string a; the result value of the insert operation is The result value of the insertion operation is used to represent the insertion of the nth character of the second string b at the mth position of the first string a; the result value of the replacement operation is , the result value of the replacement operation is used to represent that the mth character of the first string a is replaced by the nth character of the second string b. By traversing the characters in the first string a and the characters in the second string b, the category edit distance between the gesture standard category and the classification result information is calculated.

[0075] Step S300: Generate a confidence queue based on the original confidence, and generate a standard confidence of the gesture detection result according to the confidence queue.

[0076] In some embodiments, in order to reduce the jump in gesture confidence caused by changes in the shooting environment, it is necessary to perform smoothing on the gesture detection result to update the original confidence of the gesture detection result to the standard confidence. Figure 4 As shown in the figure, when the sequence value of the video frame is equal to 1, the standard confidence of the gesture detection result is the original confidence; when the sequence value of the video frame is greater than 1, the reference confidence is obtained, and a confidence queue is generated based on the reference confidence and the original confidence. Then the average of the reference confidence and the original confidence is calculated to obtain the standard confidence of the gesture detection result. Among them, the reference confidence is the original confidence of the reference frame, and the sequence value corresponding to the reference frame is less than the sequence value corresponding to the video frame.

[0077] Exemplarily, when generating a standard confidence of a video frame with a sequence value of 5, the original confidences of the video frames with sequence values ​​of 1 to 5 are obtained in turn, and a confidence queue is generated, the average value of all data in the confidence queue is calculated, and the above average value is updated to the standard confidence of the video frame with a sequence value of 5.

[0078] Exemplarily, when generating a standard confidence of a video frame with a sequence value of 6, the original confidences of the video frames with sequence values ​​of 2 to 6 are obtained in turn, and a confidence queue is generated, the average value of all data in the confidence queue is calculated, and the above average value is updated to the standard confidence of the video frame with a sequence value of 6.

[0079] It should be noted that the length of the confidence queue is an adjustable parameter, which is determined according to the actual application scenario. In addition, the standard confidence of the gesture detection result can be calculated by weighted average or median filtering. This application does not limit the calculation method of the standard confidence, as long as the confidence can be smoothed.

[0080] Step S400: Determine as target video frames those whose standard confidence is greater than or equal to a preset confidence threshold and whose similarity is greater than or equal to a preset similarity threshold; and determine as non-target video frames those whose standard confidence is less than the preset confidence threshold or whose similarity is less than the preset similarity threshold.

[0081] In some embodiments, the video frame is marked as a target video frame or a non-target video frame according to the standard confidence and similarity, wherein the target video frame is used to indicate that the video frame includes a gesture and needs to be recognized, and the non-target video frame is used to indicate that the video frame does not include a gesture and does not need to perform other processing on the video frame.

[0082] Step S5011: when the video frame is a target video frame, calculate a first gesture parameter according to a sequence value corresponding to the target video frame.

[0083] The video frame sequence is used to describe dynamic gestures. In some embodiments, the dynamic gestures include three stages, namely, the early preparation stage, the mid-term completion stage, and the late retraction stage. Among them, the early preparation stage is used to characterize the starting position, speed, and amplitude of the gesture, the mid-term completion stage is used to characterize the execution of the gesture action, and the mid-term completion stage includes complete and smooth gesture actions. The late retraction stage is used to characterize the gesture recovery process after the gesture expression is completed. It can be seen that the gesture information in the video frame corresponding to the early preparation stage is relatively vague. When the video frame corresponding to the early preparation stage is the target video frame, it is necessary to calculate the first gesture parameter according to the sequence value corresponding to the target video frame, thereby improving the credibility of gesture recognition.

[0084] In the process of calculating the first gesture parameter, a state index is first generated according to the sequence value corresponding to the target video frame, where the state index is used to represent the number of target video frames from the video frame with a sequence value of 1 to the current video frame in the video frame sequence; then the first gesture parameter is calculated using a weighted average formula, which is as follows: ; (3) in, Indicates the first gesture parameter, Indicates the status index, Represents the intensity factor, which is used to characterize the intensity of the standard confidence level corresponding to the target video frame. The intensity factor can be calculated using the intensity formula, which is as follows: ; (4) in, Indicates the preset frame number, which is determined based on prior information and can be the number of frames in which the gesture appears in the preset gesture sequence. Indicates the preset step size, which is used to represent the number of frames between the selected video frames of two adjacent operations when operating on video frames.

[0085] In this way, by calculating the first gesture parameter by a weighted average strategy, the ambiguity of the gesture in the early preparation stage can be reduced and the accuracy of gesture recognition can be improved.

[0086] Step S5012: Calculate the second gesture parameter according to the standard confidence and similarity of the target video frame.

[0087] In some embodiments, when the sequence value of the target video frame is equal to 1, the second gesture parameter is set to a first preset value; when the sequence value of the target video frame is greater than 1, the second gesture parameter is determined according to the forward confidence and the forward similarity, wherein the forward confidence is the standard confidence of the forward video frame, the forward similarity is the similarity of the forward video frame, and the forward video frame is the previous frame of the target video frame.

[0088] When the sequence value of the target video frame is greater than 1, the process of calculating the second gesture parameter includes: when the forward confidence is less than the standard confidence of the target video frame and the forward similarity is less than the similarity of the target video frame, setting the second gesture parameter to the first preset value; when the forward confidence is greater than or equal to the standard confidence of the target video frame and the forward similarity is greater than or equal to the similarity of the target video frame, setting the second gesture parameter to the second preset value.

[0089] In some embodiments, the sequence value of the target video frame is , is an integer greater than 1, and the standard confidence of the target video frame is , the similarity of the target video frame is The sequence value of the forward video frame is , the forward confidence is , the forward similarity is , then use the following formula to calculate the second gesture parameter: ; (5) in, Indicates the second gesture parameter, represents the first preset value, Indicates the second preset value.

[0090] Exemplarily, the first preset value is 1, and the second preset value is 0.

[0091] In some embodiments, the second gesture parameter can be calculated by setting the state of a single activation model. The single activation model refers to a model that generates an output result by performing an activation operation based on input data during operation. It can complete the mapping from input to output through one calculation step, thereby quickly processing and responding to the input information.

[0092] Exemplarily, the standard confidence, similarity, forward confidence and forward similarity of the target video frame are input into the single activation model, and the single activation model outputs the model state after calculation, and the model state can be an activation state and an end state. Among them, when the forward confidence is less than the standard confidence of the target video frame, and the forward similarity is less than the similarity of the target video frame, the model state output by the single activation model is the activation state. When the forward confidence is greater than or equal to the standard confidence of the target video frame, and the forward similarity is greater than or equal to the similarity of the target video frame, the model state output by the single activation model is the end state.

[0093] Then, the second gesture parameter is calculated according to the model state output by the single activation model. When the model state is the activation state, the second gesture parameter is set to the first preset value; when the model state is the end state, the second gesture parameter is set to the second preset value. For example, when the model state is the activation state, the second gesture parameter is set to 1, and when the model state is the end state, the second gesture parameter is set to 0.

[0094] In this way, the second gesture parameter is used to further characterize the similarity and confidence of the target video frame, providing a data basis for the generation of subsequent gesture recognition results.

[0095] Step S5013: Generate a gesture recognition result based on the standard confidence level, the first gesture parameter and the second gesture parameter.

[0096] In some embodiments, the gesture recognition result is used to characterize the gesture category and gesture position of the target video frame. In the process of generating the gesture result, the standard confidence, the first gesture parameter and the second gesture parameter are first multiplied to obtain a first detection result, and then the first detection result is processed by the non-maximum suppression algorithm to obtain a second detection result, and finally the second detection result is decoded to generate the gesture recognition result. Among them, the non-maximum suppression algorithm is used to screen out the final target detection result from a large number of candidate detection frames corresponding to the first detection result as the second detection result, thereby improving the accuracy and efficiency of gesture recognition.

[0097] Step S5021: When the video frame is a non-target video frame, the backward video frame is input into a gesture recognition network to perform gesture recognition on the backward video frame.

[0098] In some embodiments, a non-target video frame is used to characterize that a video frame does not include a gesture. No other processing is required for the video frame. The next frame of the non-target video frame is input into the gesture recognition network, and gesture recognition is continued on subsequent video frames in the video frame sequence until all video frames in the video frame sequence are detected to obtain a gesture recognition result.

[0099] It can be seen from the above technical solution that the present application provides a gesture recognition method, including: obtaining a video frame sequence, traversing the video frames in the video frame sequence, and inputting the video frames into a gesture recognition network to obtain gesture detection results, including gesture coordinates, gesture categories, original confidence and similarity; generating a confidence queue based on the original confidence, and generating a standard confidence of the gesture detection result based on the confidence queue; determining whether the video frame is a target video frame based on the standard confidence and similarity, and if the video frame is a target video frame, calculating a first gesture parameter based on the sequence value corresponding to the target video frame; and calculating a second gesture parameter based on the standard confidence and similarity of the target video frame; generating a gesture recognition result based on the standard confidence, the first gesture parameter and the second gesture parameter. The above method can improve the accuracy of gesture recognition and reduce the situation of gesture misdetection due to gesture change.

[0100] Based on the gesture recognition method provided in the above embodiments, some embodiments of the present application also provide a gesture recognition device, see Figure 5 The gesture recognition device includes: a video frame acquisition module 501, a gesture detection module 502, a confidence processing module 503, a video frame determination module 504 and a result output module 505.

[0101] The video frame acquisition module 501 is configured to: obtain a video frame sequence, where the video frame sequence includes video frames and sequence values ​​corresponding to the video frames, where the sequence value is a positive integer greater than or equal to 1.

[0102] The gesture detection module 502 is configured to: traverse the video frames in the video frame sequence, input the video frames into the gesture recognition network to obtain the gesture detection results corresponding to the video frames, the gesture recognition network is a deep neural network constructed based on the similarity score, and the gesture detection results include gesture coordinates, gesture category, original confidence and similarity.

[0103] The confidence processing module 503 is configured to generate a confidence queue based on the original confidence, and generate a standard confidence of the gesture detection result according to the confidence queue.

[0104] The video frame determination module 504 is configured to: determine as target video frames a video frame whose standard confidence is greater than or equal to a preset confidence threshold and whose similarity is greater than or equal to a preset similarity threshold; and determine as non-target video frames a video frame whose standard confidence is less than the preset confidence threshold or whose similarity is less than the preset similarity threshold.

[0105] The result output module 505 is configured as follows: when the video frame is a target video frame, a first gesture parameter is calculated according to a sequence value corresponding to the target video frame; and a second gesture parameter is calculated according to the standard confidence and similarity of the target video frame; a gesture recognition result is generated based on the standard confidence, the first gesture parameter and the second gesture parameter, and the gesture recognition result is used to characterize the gesture category and gesture position of the target video frame; when the video frame is a non-target video frame, a backward video frame is input into a gesture recognition network to perform gesture recognition on the backward video frame, and the backward video frame is the next frame of the non-target video frame.

[0106] Some embodiments of the present application further provide an electronic device, including: a memory and one or more processors. The memory is configured to: store one or more programs; wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the gesture recognition method.

[0107] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 6 As shown, the electronic device includes a processor 601, at least one communication bus 602, a user interface 603, at least one external communication interface 604 and a memory 605. Among them, the communication bus 602 is configured to realize the connection and communication between these components. The user interface 603 may include a display screen, and the external communication interface 604 may include a standard wired interface and a wireless interface. A computer program is stored in the memory 605. The processor 601 is used to execute the computer program stored in the memory 605.

[0108] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the general concept of this application and do not constitute a limitation on the protection scope of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without creative work belong to the protection scope of this application.

Claims

1. A gesture recognition method, characterized in that: include: Obtain a video frame sequence, wherein the video frame sequence includes video frames and sequence values ​​corresponding to the video frames, and the sequence value is a positive integer greater than or equal to 1; Traversing the video frames in the video frame sequence, and inputting the video frames into a gesture recognition network to obtain gesture detection results corresponding to the video frames, wherein the gesture recognition network is a deep neural network constructed based on a similarity score, and the gesture detection results include gesture coordinates, gesture categories, original confidence, and similarity; Generate a confidence queue based on the original confidence, and generate a standard confidence of the gesture detection result according to the confidence queue; Determine the video frame whose standard confidence is greater than or equal to a preset confidence threshold and whose similarity is greater than or equal to a preset similarity threshold as a target video frame; and determine the video frame whose standard confidence is less than the preset confidence threshold or whose similarity is less than the preset similarity threshold as a non-target video frame; In the case where the video frame is the target video frame, a first gesture parameter is calculated according to a sequence value corresponding to the target video frame; and a second gesture parameter is calculated according to the standard confidence and the similarity of the target video frame; A gesture recognition result is generated based on the standard confidence, the first gesture parameter, and the second gesture parameter, where the gesture recognition result is used to characterize the gesture category and gesture position of the target video frame.

2. The gesture recognition method according to claim 1, characterized in that: Before inputting the video frame into a gesture recognition network to obtain a gesture detection result corresponding to the video frame, the method further includes: Acquire a case data set, wherein the case data set includes at least one training gesture data, and the training gesture data is provided with a gesture annotation area and a gesture annotation category; Generate a standard gesture label according to the training gesture data, and calculate a similarity score between the training gesture data and the standard gesture label, wherein the similarity score is used to characterize the difference between the training gesture data and the standard gesture label, and the similarity score is associated with the training gesture data; Inputting the training gesture data and the similarity score into the gesture recognition network, and controlling the gesture recognition network to output regression parameters, wherein the regression parameters include coordinate information, size information, confidence information, classification result information, and similarity information; The gesture recognition network is trained based on the regression parameters, the gesture annotation categories and the loss function to optimize the model parameters of the gesture recognition network.

3. The gesture recognition method according to claim 2, characterized in that: The step of generating a standard gesture label according to the training gesture data comprises: Grouping the training gesture data according to the gesture annotation categories to obtain at least one set of gesture classification groups; Acquire a target gesture area of ​​the training gesture data according to the gesture annotation area, and input the target gesture area into a feature extraction model to obtain gesture features, wherein the gesture features are associated with the training gesture data; A standard gesture label is generated according to the gesture feature, and the standard gesture label is used to characterize a reference gesture of the standard gesture category.

4. The gesture recognition method according to claim 3, characterized in that: The training gesture data includes continuous frame data and discontinuous frame data; In a case where the training gesture data is the continuous frame data, marking the gesture features of the training gesture data as continuous frame gesture features; In the case where the training gesture data is the non-continuous frame data, a preset weight is multiplied by the gesture feature of the training gesture data to obtain a non-continuous frame gesture feature.

5. The gesture recognition method according to claim 4, characterized in that: The step of generating a standard gesture label according to the gesture feature comprises: Acquire reference gesture features of the training gesture data in each of the gesture category groups, the reference gesture features including the continuous frame gesture features and / or the non-continuous frame gesture features; The average value of the reference gesture features is calculated to obtain the standard gesture label.

6. The gesture recognition method according to claim 3, characterized in that: The calculating the similarity score between the training gesture data and the standard gesture label comprises: Acquire reference gesture data, where the reference gesture data is a standard gesture label of the gesture classification group where the training gesture data belongs; A cosine distance between the training gesture data and the reference gesture data is calculated to obtain the similarity score.

7. The gesture recognition method according to claim 3, characterized in that: The loss function includes class loss, target loss, position loss and similarity loss, wherein the similarity loss is the quotient of the class edit distance and the number of classes, wherein the class edit distance is used to characterize the difference between the classification result information and the gesture annotation class, and the number of classes is used to characterize the number of the gesture classification groups; The step of training the gesture recognition network based on the regression parameter, the gesture annotation category and the loss function to optimize the model parameters of the gesture recognition network includes: Optimizing classification parameters of the gesture recognition network based on the class loss, wherein the classification parameters are used to calculate the classification result information; Optimizing a confidence parameter of the gesture recognition network based on the target loss, wherein the confidence parameter is used to calculate the confidence information; Optimizing position parameters of the gesture recognition network based on the position loss, wherein the position parameters are used to calculate the coordinate information, width information, and height information; A similarity parameter of the gesture recognition network is optimized based on the similarity loss, and the similarity parameter is used to calculate the similarity information.

8. The gesture recognition method according to claim 1, characterized in that: Generating a confidence queue based on the original confidence, and generating a standard confidence of the gesture detection result according to the confidence queue, including: When the sequence value of the video frame is equal to 1, the standard confidence of the gesture detection result is the original confidence; When the sequence value of the video frame is greater than 1, a reference confidence is obtained, where the reference confidence is the original confidence of the reference frame, and the sequence value corresponding to the reference frame is less than the sequence value corresponding to the video frame; and the confidence queue is generated according to the reference confidence and the original confidence, and the average value of the reference confidence and the original confidence is calculated to obtain the standard confidence of the gesture detection result.

9. The gesture recognition method according to claim 1, characterized in that: Also includes: In a case where the video frame is the non-target video frame, a backward video frame is input into the gesture recognition network to perform gesture recognition on the backward video frame, where the backward video frame is a next frame of the non-target video frame.

10. The gesture recognition method according to claim 1, characterized in that: The calculating the first gesture parameter according to the sequence value corresponding to the target video frame includes: Generate a state index according to the sequence value corresponding to the target video frame, where the state index is used to represent the number of target video frames between the video frame with the sequence value of 1 and the current video frame in the video frame sequence; The first gesture parameter is calculated using a weighted average formula, where the weighted average formula is: ， in, represents the state index, represents the first gesture parameter, represents the intensity factor, which is calculated using the intensity formula, which is: ， in, Indicates the preset frame number. Indicates the preset step size.

11. The gesture recognition method according to claim 1, characterized in that: The calculating the second gesture parameter according to the standard confidence and the similarity comprises: When the sequence value of the target video frame is equal to 1, setting the second gesture parameter to a first preset value; When the sequence value of the target video frame is greater than 1, a forward confidence and a forward similarity are obtained, wherein the forward confidence is a standard confidence of the forward video frame, the forward similarity is a similarity of the forward video frame, and the forward video frame is a previous frame of the target video frame; When the forward confidence is less than the standard confidence of the target video frame, and the forward similarity is less than the similarity of the target video frame, setting the second gesture parameter to a first preset value; When the forward confidence is greater than or equal to the standard confidence of the target video frame, and the forward similarity is greater than or equal to the similarity of the target video frame, the second gesture parameter is set to a second preset value.

12. The gesture recognition method according to claim 1, characterized in that: The generating a gesture recognition result based on the standard confidence, the first gesture parameter and the second gesture parameter includes: multiplying the standard confidence, the first gesture parameter, and the second gesture parameter to obtain a first detection result; Performing non-maximum suppression algorithm processing on the first detection result to obtain a second detection result; The second detection result is decoded to generate the gesture recognition result.

13. A gesture recognition device, characterized in that: include: A video frame acquisition module is configured to: obtain a video frame sequence, wherein the video frame sequence includes video frames and sequence values ​​corresponding to the video frames, and the sequence value is a positive integer greater than or equal to 1; A gesture detection module is configured to: traverse the video frames in the video frame sequence, input the video frames into a gesture recognition network to obtain a gesture detection result corresponding to the video frame, wherein the gesture recognition network is a deep neural network constructed based on a similarity score, and the gesture detection result includes gesture coordinates, gesture category, original confidence and similarity; A confidence processing module is configured to: generate a confidence queue based on the original confidence, and generate a standard confidence of the gesture detection result according to the confidence queue; The video frame determination module is configured to: determine the video frame whose standard confidence is greater than or equal to a preset confidence threshold and whose similarity is greater than or equal to a preset similarity threshold as a target video frame; and determine the video frame whose standard confidence is less than the preset confidence threshold or whose similarity is less than the preset similarity threshold as a non-target video frame; The result output module is configured to: when the video frame is the target video frame, calculate the first gesture parameter according to the sequence value corresponding to the target video frame; and calculate the second gesture parameter according to the standard confidence and the similarity of the target video frame; generating a gesture recognition result based on the standard confidence, the first gesture parameter, and the second gesture parameter, wherein the gesture recognition result is used to characterize the gesture category and gesture position of the target video frame; In a case where the video frame is the non-target video frame, a backward video frame is input into the gesture recognition network to perform gesture recognition on the backward video frame, where the backward video frame is a next frame of the non-target video frame.

14. An electronic device, characterized in that: include: memory, one or more processors; The memory is configured to: store one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the gesture recognition method according to any one of claims 1-12.

Images