Model training method, gesture recognition method, device, apparatus and storage medium

Abstract

Embodiments of the present application disclose a model training method, a gesture recognition method, an apparatus, a device and a storage medium. The method comprises: obtaining at least one gesture picture and at least one non-gesture picture; performing picture fusion processing on each of the at least one gesture picture and the at least one non-gesture picture to obtain a sample picture set; and training a preset network model by using the sample picture set to obtain a gesture recognition model. The gesture recognition model comprises a first sub-model and a second sub-model. The first sub-model is used to determine a gesture type of a to-be-detected picture, and the second sub-model is used to determine hand positioning information of the to-be-detected picture. In this way, the gesture recognition model obtained by training according to the sample picture set, which comprises not only gestures but also non-gesture content, is more suitable for a real scene, and the recognition accuracy and model robustness of the gesture recognition model are enhanced, and the interference of external factors on the recognition result is reduced.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction, and in particular to a model training method, a gesture recognition method, a device, an apparatus, and a storage medium. Background Technology

[0002] Gesture recognition technology provides a natural and intuitive way for people to communicate with terminal devices. However, current gesture recognition technologies typically rely on skin color segmentation to locate the hand and manual extraction of gesture features. These methods are highly susceptible to environmental influences, resulting in low accuracy. Summary of the Invention

[0003] This application provides a model training method, a gesture recognition method, a device, an apparatus, and a storage medium, which can obtain a gesture recognition model that is more suitable for real-world scenarios, improve the accuracy of gesture recognition, and reduce interference from external factors such as lighting.

[0004] The technical solution of this application is implemented as follows:

[0005] In a first aspect, embodiments of this application provide a model training method, the method comprising:

[0006] Obtain at least one gesture image and at least one non-gesture image;

[0007] Each gesture image in the at least one gesture image is fused with the at least one non-gesture image to obtain a sample image set;

[0008] The preset network model is trained using the sample image set to obtain a gesture recognition model; wherein, the gesture recognition model includes a first sub-model and a second sub-model, the first sub-model is used to determine the gesture type of the image to be detected, and the second sub-model is used to determine the hand positioning information of the image to be detected.

[0009] Secondly, embodiments of this application provide a gesture recognition method applied to the gesture recognition model as described in the first aspect, the method comprising:

[0010] Obtain the video stream to be detected;

[0011] The gesture recognition model is used to perform gesture detection on each video frame in the video stream to be detected, and the gesture type and hand positioning information in each video frame are determined.

[0012] The hand location information in the video stream to be detected is subjected to overlap (IOU) detection to determine the hand detection result of the video stream to be detected;

[0013] When the hand detection result indicates that the hand in the video stream to be detected is the same hand, the gesture type recognition result of the same hand in the video stream to be detected is determined according to the gesture type in each video frame.

[0014] Thirdly, embodiments of this application provide a model training apparatus, which includes a first acquisition unit, a fusion unit, and a training unit, wherein...

[0015] The first acquisition unit is configured to acquire at least one gesture image and at least one non-gesture image;

[0016] The fusion unit is configured to perform image fusion processing on each gesture image in the at least one gesture image and the at least one non-gesture image to obtain a sample image set;

[0017] The training unit is configured to train a preset network model using the sample image set to obtain a gesture recognition model; wherein the gesture recognition model includes a first sub-model and a second sub-model, the first sub-model is used to determine the gesture type of the image to be detected, and the second sub-model is used to determine the hand positioning information of the image to be detected.

[0018] Fourthly, embodiments of this application provide a gesture recognition device, which includes a second acquisition unit, a gesture detection unit, an IOU detection unit, and a determination unit, wherein...

[0019] The second acquisition unit is configured to acquire the video stream to be detected;

[0020] The gesture detection unit is configured to use the gesture recognition model to perform gesture detection on each video frame in the video stream to be detected, and to determine the gesture type and hand positioning information in each video frame.

[0021] The IOU detection unit is configured to perform overlap IOU detection on the hand positioning information in the video stream to be detected, and determine the hand detection result of the video stream to be detected.

[0022] The determining unit is configured to determine the gesture type recognition result of the same hand in the video stream to be detected based on the gesture type in each video frame when the hand detection result indicates that the hand in the video stream to be detected is the same hand.

[0023] Fifthly, embodiments of this application provide an electronic device, which includes a memory and a processor, wherein,

[0024] The memory is used to store computer programs that can run on the processor;

[0025] The processor is configured to, when running the computer program, execute the model training method as described in the first aspect; or execute the gesture recognition method as described in the second aspect.

[0026] In a sixth aspect, embodiments of this application provide a computer storage medium storing a computer program that, when executed by a processor, implements the model training method as described in the first aspect; or, implements the gesture recognition method as described in the second aspect.

[0027] This application provides a model training method, gesture recognition method, apparatus, device, and storage medium. During model training, at least one gesture image and at least one non-gesture image are acquired. Each gesture image in the at least one gesture image is fused with at least one non-gesture image to obtain a sample image set. A preset network model is trained using the sample image set to obtain a gesture recognition model. The gesture recognition model includes a first sub-model and a second sub-model. The first sub-model determines the gesture type of the image to be detected, and the second sub-model determines the hand location information of the image to be detected. During gesture recognition, a video stream to be detected is acquired. The gesture recognition model performs gesture detection on each video frame in the video stream to determine the gesture type and hand location information in each video frame. Intersection over Union (IOU) detection is performed on the hand location information in the video stream to determine the hand detection result. When the hand detection result indicates that the hand in the video stream is the same hand, the gesture type recognition result for the same hand in the video stream is determined based on the gesture type in each video frame. In this way, since the sample images include not only gestures but also non-gesture content, the gesture recognition model trained based on this is more applicable to real-world scenarios, enhancing the recognition accuracy and robustness of the gesture recognition model and reducing the interference of external factors on the recognition results. In addition, by performing gesture detection on each video frame in the video stream to be detected, and tracking the hand in the video stream to be detected based on the detected hand positioning information, the same hand in the video stream to be detected is identified, and the gesture type recognition result of the same hand in the video stream to be detected is further determined based on the detected gesture type; thus improving the accuracy and precision of gesture recognition. Attached Figure Description

[0028] Figure 1 A schematic flowchart illustrating a model training method provided in an embodiment of this application;

[0029] Figure 2 A schematic diagram illustrating an image fusion process provided in an embodiment of this application;

[0030] Figure 3 A flowchart illustrating a gesture recognition method provided in an embodiment of this application;

[0031] Figure 4 This is a schematic diagram of a gesture recognition result for an image provided in an embodiment of this application;

[0032] Figure 5 This is a schematic diagram of the architecture of a gesture recognition device provided in an embodiment of this application;

[0033] Figure 6 A detailed flowchart illustrating a gesture recognition method provided in an embodiment of this application;

[0034] Figure 7 This application provides a schematic diagram of hierarchical fusion of a network structure.

[0035] Figure 8 This is a schematic diagram of the composition structure of a preset network model provided in an embodiment of this application;

[0036] Figure 9 A schematic diagram illustrating the composition structure of a classification branch network and a regression branch network provided in an embodiment of this application;

[0037] Figure 10 This is a schematic diagram of the composition structure of a model training device provided in an embodiment of this application;

[0038] Figure 11 This is a schematic diagram of the composition structure of a gesture recognition device provided in an embodiment of this application;

[0039] Figure 12 This is a schematic diagram of the specific hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0040] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the relevant application and not for limiting the application. Furthermore, it should be noted that, for ease of description, only the parts related to the relevant application are shown in the accompanying drawings.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0042] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0043] It should be noted that the terms "first, second, and third" used in the embodiments of this application are merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, and third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0044] Understandably, gesture recognition technology provides a natural and intuitive way for people to communicate with terminal devices. In real-world scenarios, gesture recognition mainly involves multiple processes, including hand detection, gesture feature extraction, and classification. Currently, hand detection technologies primarily include skin color segmentation-based hand detection methods and deep learning-based hand detection methods; gesture feature extraction and classification are mainly divided into traditional hand-designed feature extraction and classification and deep learning-based methods.

[0045] Currently, a method for gesture recognition based on the YOLO (You Only Look Onece) network has been proposed. This method uses a simple threshold skin color segmentation algorithm based on the YCbCr color space to filter the background of the image to be recognized before determining the gesture recognition result. However, this method is not effective at filtering skin-colored backgrounds and body parts such as the face, and is easily affected by the surrounding environment, resulting in poor robustness of the gesture recognition algorithm. In addition, the YOLO model runs slowly, leading to poor real-time performance; moreover, this method only supports single-hand gesture recognition.

[0046] In addition, related technologies have proposed a computer vision-based gesture recognition method, which locates the fingertips of the hand. However, in real-world scenarios, fingertips are often occluded, which greatly limits the applicability and detection accuracy of the model. Moreover, this method requires loading two deep learning models (a hand detection model and a gesture recognition recurrent neural network model) for gesture recognition, increasing the algorithm's processing time, severely impacting its efficiency, and wasting computational resources. Related technologies have also proposed a deep learning-based gesture recognition method, using an improved YOLOv3 model to complete the gesture recognition task. However, on the one hand, the YOLO model runs slowly; on the other hand, this model is easily affected by background, lighting, and other factors, resulting in poor model accuracy. Related technologies have also proposed a gesture interaction method. This method captures the user's gestures with a camera and uses a pre-trained deep neural network to perform hand detection and gesture recognition on the captured images. After obtaining the first gesture recognition result, it analyzes and confirms the gesture based on the first gesture recognition result in multiple frames of images to obtain the second gesture recognition result, and triggers the execution of the operation corresponding to the second gesture recognition result to complete the gesture interaction. However, this method requires loading three algorithm models simultaneously (hand detection algorithm model, tracking algorithm model, and gesture recognition algorithm model), which results in long processing time, low running efficiency, and wasted computing resources. Moreover, this method requires hardware devices such as depth image cameras to obtain depth image information, which greatly limits its application scenarios.

[0047] In other words, current gesture recognition methods require hand localization based on skin color segmentation and manual extraction of gesture features, which are easily affected by the environment. Hand localization and feature design and selection have a significant impact on the gesture recognition results. The rapid development of deep learning and its application in the field of gesture recognition have greatly improved the accuracy of gesture recognition, but the following problems still exist: (1) Due to the complexity of deep learning models, the real-time performance of gesture recognition is poor; (2) Existing static gesture recognition technology often extracts and recognizes features from specific video frames in isolation, losing the temporal information between video frames, which leads to a decrease in the accuracy of gesture recognition; (3) In existing gesture recognition methods, in order to consider the temporal features of gestures in previous and subsequent frames for multi-hand gesture recognition, target tracking algorithms are often introduced, which not only increases the spatial and temporal redundancy of the gesture recognition algorithm, but also wastes computational resources; (4) There are few public gesture datasets available at present, and their application scenarios are different, so the training set of gesture recognition algorithms often comes from self-made datasets. However, due to environmental and other constraints, self-made datasets have disadvantages such as single background and single illumination, resulting in poor robustness of the obtained algorithm model.

[0048] Based on this, this application provides a model training method. The basic idea of ​​this method is: to acquire at least one gesture image and at least one non-gesture image; to perform image fusion processing on each gesture image in the at least one gesture image and at least one non-gesture image to obtain a sample image set; to train a preset network model using the sample image set to obtain a gesture recognition model; wherein, the gesture recognition model includes a first sub-model and a second sub-model, the first sub-model is used to determine the gesture type of the image to be detected, and the second sub-model is used to determine the hand positioning information of the image to be detected.

[0049] This application also provides a gesture recognition method. The basic idea of ​​this method is as follows: acquire a video stream to be detected; use a gesture recognition model to perform gesture detection on each video frame in the video stream to be detected, and determine the gesture type and hand positioning information in each video frame; perform overlap (IOU) detection on the hand positioning information in the video stream to be detected, and determine the hand detection result in the video stream to be detected; when the hand detection result indicates that the hand in the video stream to be detected is the same hand, determine the gesture type recognition result of the same hand in the video stream to be detected based on the gesture type in each video frame.

[0050] In this way, since the sample images include not only gestures but also non-gesture content, the gesture recognition model trained based on this is more applicable to real-world scenarios, enhancing the recognition accuracy and robustness of the model and reducing the interference of external factors on the recognition results. Furthermore, by performing gesture detection on each frame of the video stream to be detected, and tracking the hands in the video stream based on the detected hand positioning information, the same hand in the video stream is identified, and the gesture type recognition result of the same hand in the video stream is further determined based on the detected gesture type; thus, the accuracy and precision of gesture recognition are improved.

[0051] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0052] In one embodiment of this application, see [link to embodiment]. Figure 1 This illustrates a flowchart of a model training method provided in an embodiment of this application. Figure 1 As shown, the method may include:

[0053] S101. Obtain at least one gesture image and at least one non-gesture image.

[0054] S102. Perform image fusion processing on each gesture image in at least one gesture image and at least one non-gesture image to obtain a sample image set.

[0055] It should be noted that the model training method provided in this application embodiment can be applied to a model training device or an electronic device integrated with such a device. Here, the electronic device can be such as a computer, smartphone, tablet computer, laptop computer, handheld computer, personal digital assistant (PDA), navigation device, server, etc., and this application embodiment does not specifically limit it in this regard.

[0056] It should also be noted that gesture images can include images containing gestures captured in real-world scenarios, as well as gesture images obtained from sources such as films and television shows, and may even include gesture images of cartoon characters. This application does not specifically limit these.

[0057] Non-gesture images can include real-world scene images, landscape images, etc., or various images from non-real-world scenes, and this application embodiment does not specifically limit them.

[0058] It should also be noted that the embodiments of this application obtain a sample image set by fusing gesture images and non-gesture images. Here, a sample image in the sample image set can be a sample image obtained by fusing a gesture image with a non-gesture image, or it can be a sample image obtained by fusing a gesture image with several non-gesture images. In this way, for a single gesture image, it can be fused with multiple non-gesture images, thereby expanding a single gesture image into multiple sample images and enriching the sample image set.

[0059] In this way, on the one hand, the sample images contain not only gesture content but also non-gesture content, which reduces the impact of environmental factors on the model recognition results after model training; on the other hand, it can enrich the sample image set when the gesture images are not rich enough.

[0060] In some embodiments, the image fusion process may include at least: image fusion processing and / or image stitching processing.

[0061] It should be noted that when performing image fusion processing on gesture images and non-gesture images, the gesture image can be merged and / or stitched together with the non-gesture image. For example, see... Figure 2 This illustration shows a schematic diagram of an image fusion processing procedure provided in an embodiment of this application. Figure 2 In this process, a landscape image and a gesture image can be fused together to obtain a fused sample image.

[0062] Furthermore, in some embodiments, the method may further include: adding annotation information to sample images in the sample image set; wherein the annotation information may include gesture type information and hand positioning information in the sample images, and the hand positioning information may include the center coordinates of the hand positioning box and the width and height of the hand positioning box.

[0063] It should be noted that the embodiments of this application can add annotation information to the sample image. This annotation information may include the gesture type information and hand positioning information of the hand contained in the sample image. Specifically, the hand positioning information may include the center coordinates of the hand positioning frame and the width and height of the hand positioning frame.

[0064] Furthermore, in this embodiment, annotation information can be added to the gesture image before image fusion, and then the gesture image with the added annotation information can be fused with the non-gesture image. In this way, the fused sample image also has the annotation information; this embodiment does not specifically limit this.

[0065] It should also be noted that when adding annotation information, image annotation tools disclosed in the art, such as VGG Image Annotator (abbreviated as VIA), can be used. However, the embodiments of this application do not specifically limit the image annotation tools used.

[0066] S103. Train the preset network model using the sample image set to obtain the gesture recognition model.

[0067] It should be noted that the gesture recognition model can include a first sub-model and a second sub-model. The first sub-model can be used to determine the gesture type of the image to be detected, and the second sub-model can be used to determine the hand positioning information of the image to be detected.

[0068] In other words, the gesture recognition model obtained by training a preset network model using a set of sample images can not only determine the gesture type of the image to be detected, but also determine the hand positioning information of the image to be detected.

[0069] Furthermore, to improve the accuracy of the gesture recognition model, the embodiments of this application may preprocess the sample images before training the preset network model using the sample image set. Therefore, in some embodiments, the method may further include:

[0070] Resize the sample images in the sample image set; and / or,

[0071] Perform color mode conversion on the sample images in the sample image set.

[0072] It should be noted that size adjustment can include adjusting all sample images to a preset size, for example, a pixel size of 320×320; color mode conversion can include converting all sample images to a preset color mode, for example, a red-green-blue color mode (RGB color mode).

[0073] When a preset network model is trained using a sample image set to obtain a gesture recognition model, in some embodiments, the preset network model may include a feature extraction network, a classification branch network, and a regression branch network; correspondingly, training the preset network model using the sample image set to obtain the gesture recognition model may include:

[0074] Input the sample images from the sample image set into the preset network model;

[0075] The feature extraction network is used to process the sample images in the sample image set to obtain at least one target feature map;

[0076] The classification branch network is trained using at least one target feature map to obtain the first sub-model; and the regression branch network is trained using at least one target feature map to obtain the second sub-model.

[0077] Based on the first and second sub-models, the gesture recognition model is determined.

[0078] It should be noted that when training the preset network model, sample images from the sample image set can be input into the preset network model sequentially or simultaneously, and at least one target feature map corresponding to the sample image can be obtained through the feature extraction network of the preset network model. Then, the classification branch network and regression branch network of the preset network model are trained using these at least one target feature map respectively to obtain the first sub-model and the second sub-model, and the gesture recognition model is determined based on the first sub-model and the second sub-model.

[0079] Furthermore, for the feature extraction network, in some embodiments, the feature extraction network includes a feature extraction layer, a feature fusion layer, and a feature convolutional layer; the step of performing feature processing on sample images in the sample image set through the feature extraction network to obtain at least one target feature map may include:

[0080] The initial feature extraction layer is used to extract initial features from the sample image to determine at least one initial feature map corresponding to the sample image.

[0081] The feature fusion layer is used to perform feature interaction fusion on at least one initial feature map to obtain at least one fused feature map corresponding to the sample image;

[0082] At least one fused feature map is convolved using a feature convolutional layer to obtain at least one target feature map.

[0083] It should be noted that when determining the target feature map, for any sample image, the feature extraction layer of the feature extraction network is first used to perform initial feature extraction on the sample image at at least one feature layer, thereby obtaining at least one initial feature map corresponding to the sample image.

[0084] It should also be noted that the preset network model includes the SSD_mobilenetV3_I model. In some embodiments, the feature extraction layer may include feature layers Layer14, Layer17, Layer20, Layer23, Layer26, and Layer29 in the SSD_mobilenetV3_I model.

[0085] For example, the feature extraction layer includes six feature layers: Layer14, Layer17, Layer20, Layer23, Layer26, and Layer29. After the initial feature extraction, a sample image can obtain its corresponding initial feature map in each of these six feature layers.

[0086] Furthermore, the feature fusion layer of the feature extraction network is used to perform feature cross-fusion on at least one initial feature map, thereby obtaining at least one fused feature map corresponding to the sample image. Taking the SSD_mobilenetV3_I model mentioned above as an example, for the six feature layers, Layer14, Layer17, Layer20, Layer23, Layer26, and Layer29, six fused feature maps can be obtained respectively. Here, the feature fusion layer may include a concatenated layer, that is, the six initial feature maps of the sample image are fused in a concatenated manner to obtain six fused feature maps with rich features.

[0087] Furthermore, the feature extraction network's feature convolutional layers are used to perform convolution operations on at least one feature fusion map, ultimately obtaining at least one target feature map. Taking the SSD_mobilenetV3_I model as an example, performing convolution operations on the six obtained feature fusion maps respectively yields six corresponding target feature maps. Here, the feature convolutional layers may include convolutional layers with a kernel size of 1×1 (denoted as Conv1×1).

[0088] It should also be noted that, for the fused feature map, in some embodiments, the step of using the feature fusion layer to perform feature interaction fusion on at least one initial feature map to obtain at least one fused feature map corresponding to the sample image may include:

[0089] Determine the initial feature map of the sample image in the i-th feature layer and the initial feature maps of other feature layers besides the i-th feature layer;

[0090] The initial feature maps of all feature layers except the i-th feature layer are sampled to obtain the sampling results.

[0091] The initial feature map and sampling results of the i-th feature layer are fused using a feature fusion layer to obtain the fused feature map of the sample image at the i-th feature layer; where i is an integer greater than zero.

[0092] It should be noted that when using the feature fusion layer to perform feature interaction fusion on the initial feature map to obtain the fused feature map, the feature fusion layer fuses the initial feature map corresponding to each feature layer with the sampling results of the initial feature maps of other feature layers.

[0093] In other words, for the i-th feature layer, it is necessary to sample the initial feature maps of other feature layers (excluding the i-th feature layer) to obtain the sampling results. The sampling results corresponding to the initial feature maps of other feature layers are then fused with the initial feature map of the i-th feature layer through the feature fusion layer, thereby obtaining the fused feature map corresponding to the i-th feature layer of the sample image.

[0094] Taking the feature layer Layer14 in the SSD_mobilenetV3_I model mentioned above as an example, the feature layers Layer17, Layer20, Layer23, Layer26 and Layer29 are sampled respectively to obtain five sampling results. Then, these five sampling results are fused with the initial feature map of Layer14 to obtain the fused feature map corresponding to Layer14.

[0095] In a specific example, the other feature layers besides the i-th feature layer may include: a first feature layer portion located before the i-th feature layer and a second feature layer portion located after the i-th feature layer, wherein the level of the first feature layer portion is higher than the level of the i-th feature layer, and the level of the i-th feature layer is higher than the level of the second feature layer portion.

[0096] Accordingly, the sampling process for the initial feature maps of feature layers other than the i-th feature layer to obtain the sampling results may include:

[0097] The initial feature map of the first feature layer is downsampled and the initial feature map of the second feature layer is upsampled to obtain the sampling result.

[0098] It should be noted that for the i-th feature layer, the other feature layers can include two parts: the first feature layer part located before the i-th feature layer, that is, the part with a higher level than the i-th feature layer, and the second feature layer part located after the i-th feature layer, that is, the part with a lower level than the i-th feature layer.

[0099] It should also be noted that, regarding the sampling process, the embodiments of this application perform downsampling on the initial feature map of the first feature layer, while performing upsampling on the initial feature map of the second feature layer.

[0100] For example, taking feature layer 20 in the SSD_mobilenetV3_I model above, the first feature layer preceding layer 20 includes Layer 14 and Layer 17, and the second feature layer following layer 20 includes Layer 23, Layer 26, and Layer 29. Layer 14 and Layer 17 are downsampled, and Layer 23, Layer 26, and Layer 29 are upsampled. Alternatively, taking feature layer 14 in the SSD_mobilenetV3_I model above, there is no first feature layer preceding layer 14, and the second feature layer following layer 14 includes Layer 17, Layer 20, Layer 23, Layer 26, and Layer 29. Layer 17, Layer 20, Layer 23, Layer 26, and Layer 29 are upsampled. Taking feature layer Layer29 in the SSD_mobilenetV3_I model above as an example, the first feature layer preceding Layer29 includes Layer14, Layer17, Layer20, Layer23, and Layer26, and there is no second feature layer after it. Therefore, downsampling is performed on Layer14, Layer17, Layer20, Layer23, and Layer26 respectively.

[0101] In this way, by sampling and then fusing the initial feature maps corresponding to other feature layers besides the i-th feature layer, the interactive fusion of contour detail features of the low-level and middle-level feature layers with semantic features of the high-level feature layers is achieved. Through this parallel and independent feature fusion method, each feature layer can simultaneously possess high-level semantic features and low-level and middle-level contour detail features.

[0102] In some embodiments, downsampling can be performed using a max pooling function, and upsampling can be performed using a deconvolution function.

[0103] In other words, downsampling can be achieved using the max pooling function, while upsampling can be achieved using the deconvolution function.

[0104] Furthermore, for the first sub-model, the classification branch network may include an attention mechanism module; wherein, the attention mechanism module may include a convolutional sub-module, an activation sub-module, a multiplication sub-module, and an addition sub-module, and the convolutional sub-module consists of convolutional layers, transposed convolutional layers, and skip layers. Correspondingly, in some embodiments, training the classification branch network using at least one target feature map to obtain the first sub-model may include:

[0105] Determine the loss function of the first model, and use a classification branch network including an attention mechanism module to classify and train at least one target feature map. When the value of the loss function of the first model reaches a preset convergence value, the trained classification branch network is determined as the first sub-model.

[0106] It should be noted that the embodiments of this application introduce an attention mechanism module into the classification branch network, which enables the classification branch network to pay more attention to the gesture-related features of the sample image, that is, the classification branch network pays more attention to semantic features.

[0107] Here, the attention mechanism module can include a convolution submodule, an activation submodule, a multiplication submodule, and an addition submodule. The convolution submodule further extracts features from a target feature map, the result of which can be called the initial classification feature matrix. Then, the activation submodule estimates the probability of the features in the initial classification feature matrix to obtain a classification probability matrix. Features with a higher correlation to the gesture type have higher probability values. For example, the activation function of the activation submodule can be the sigmoid function. Due to its monotonically increasing nature and the monotonically increasing nature of its inverse function, the sigmoid function can map features to the range [0,1]. The multiplication submodule performs element-wise product processing on the classification probability matrix and the target feature map to obtain a classification product matrix. Since the classification probability matrix represents the probability values ​​of the features, features with a higher correlation to the gesture type are amplified after element-wise product with the target feature map. The addition submodule performs element-wise sum processing on the classification product matrix and the target feature map to obtain a classification sum matrix.

[0108] The convolutional submodule can include convolutional layers, transposed convolutional layers, and skip layers. The convolutional layers can include convolutional layers with a kernel size of 1×1 (represented by Conv1×1) and convolutional layers with a kernel size of 3×3 (represented by Conv3×3). The transposed convolutional layers include transposed convolutional layers with a kernel size of 3×3 (represented by Deconv3×3). The skip layers are used to perform skip connections on the results of convolutional processing of the convolutional layers and the transposed convolutional layers.

[0109] In the attention mechanism module of the classification branch network, by processing each target feature map of the sample image in this way, at least one classification summation matrix can be obtained. For at least one classification summation matrix, the classification branch network can use the non-maximum suppression (NMS) algorithm to remove duplicates and predict the gesture type of the corresponding sample image. The predicted gesture type is compared with the real gesture type, and the value of the first model loss function is determined. This process continues until the value of the first model loss function reaches a preset convergence value, that is, the model error value is small, or the model error value will not decrease after further training. This indicates that the model has converged. At this point, the trained classification branch network is determined as the first sub-model, and the first sub-model is obtained.

[0110] It should also be noted that the loss function of the first model can be a loss function commonly used in the art, such as the absolute value loss function, the 0-1 loss function, the Hinge loss function, the perceptual loss function, or the cross-entropy loss function, etc., and the embodiments of this application do not specifically limit it.

[0111] Furthermore, for the second sub-model, the regression branch network may include an attention mechanism module; wherein, the attention mechanism module may include a convolutional sub-module, an activation sub-module, a multiplication sub-module, and an addition sub-module, and the convolutional sub-module consists of convolutional layers, transposed convolutional layers, and skip layers. Correspondingly, in some embodiments, training the regression branch network using at least one target feature map to obtain the second sub-model may include:

[0112] Determine the loss function of the second model, and use a regression branch network including an attention mechanism module to perform regression training on at least one target feature map. When the value of the loss function of the second model reaches the preset convergence value, the trained regression branch network is determined as the second sub-model.

[0113] It should be noted that the embodiments of this application introduce an attention mechanism module in the regression branch network, which enables the regression branch network to pay more attention to the features of the sample image related to the hand positioning information, that is, the regression branch network pays more attention to the contour detail features.

[0114] Here, the attention mechanism module can still include a convolution submodule, an activation submodule, a multiplication submodule, and an addition submodule. The convolution submodule further extracts features from a target feature map, the result of which can be called the initial regression feature matrix. Then, the activation submodule estimates the probability of the features in the initial regression feature matrix to obtain a regression probability matrix. Features with a higher correlation to hand positioning information have higher probability values. For example, the activation function of the activation submodule can still be the sigmoid function, used to map features to the range [0,1]. The multiplication submodule performs element-wise multiplication of the regression probability matrix with the target feature map to obtain a regression product matrix. Since the regression probability matrix represents the probability value of a feature, features with a higher correlation to hand positioning information are amplified after element-wise multiplication with the target feature map. The addition submodule performs element-wise addition of the regression product matrix with the target feature map to obtain a regression summation matrix.

[0115] The convolutional submodule can include convolutional layers, transposed convolutional layers, and skip layers. The convolutional layers include convolutional layers with a kernel size of 1×1 and convolutional layers with a kernel size of 3×3. The transposed convolutional layers include transposed convolutional layers with a kernel size of 3×3. The skip layers are used to connect the results of convolution processing of the convolutional layers and the transposed convolutional layers.

[0116] In the attention mechanism module of the regression branch network, each target feature map of the sample image is processed in this way to obtain at least one regression summation matrix. For at least one regression summation matrix, the regression branch network can use a non-maximum suppression algorithm to remove duplicates and predict the hand localization information of the corresponding sample image. The predicted hand localization information is compared with the true hand localization information, and the value of the second model loss function is determined. This process continues until the value of the second model loss function reaches a preset convergence value, that is, the model error is small, or the model error will not decrease after further training. This indicates that the model has converged. At this point, the trained regression branch network is determined as the second sub-model, and the second sub-model is obtained.

[0117] It should also be noted that the loss function of the second model can also be a loss function commonly used in the art, such as the absolute value loss function, the 0-1 loss function, the Hinge loss function, the perceptual loss function, or the cross-entropy loss function, etc. The embodiments of this application do not specifically limit this.

[0118] In addition, it should be noted that since the initial feature maps of sample images in different feature layers are fused in the aforementioned feature extraction network, although a richer target feature map can be obtained, it also leads to feature redundancy. Therefore, amplifying the relevant features in the attention mechanism modules of the classification branch network and the regression branch network can also play a role in removing redundancy.

[0119] This embodiment provides a model training method, which involves acquiring at least one gesture image and at least one non-gesture image; performing image fusion processing on each gesture image and at least one non-gesture image to obtain a sample image set; and training a preset network model using the sample image set to obtain a gesture recognition model. The gesture recognition model includes a first sub-model and a second sub-model, wherein the first sub-model is used to determine the gesture type of the image to be detected, and the second sub-model is used to determine the hand positioning information of the image to be detected. In this way, since the sample images in the sample image set are obtained by image fusion processing of gesture images and non-gesture images, the sample images not only include gestures but also non-gesture content. The gesture recognition model trained with such a sample image set is more suitable for real-world scenarios, enhancing the recognition accuracy and robustness of the gesture recognition model and reducing interference from external factors such as lighting. Moreover, image fusion processing of gesture images can enrich the sample image set to the greatest extent when there are limited gesture images. In addition, during the model training process, the target feature map incorporates multiple layers of features when extracting features from the sample images, making the features in the target feature map richer, possessing both high-level semantic features and low- and mid-level contour detail features. Furthermore, since attention mechanism modules are introduced into both the classification branch network and the regression branch network, the trained first sub-model focuses more on features related to gesture type, and the trained second sub-model focuses more on features related to hand positioning information, further improving the accuracy of the gesture recognition model in recognizing gesture types and determining hand positioning information.

[0120] In another embodiment of this application, see Figure 3 This illustration shows a flowchart of a gesture recognition method provided in an embodiment of this application. This method can apply the gesture recognition model described in the foregoing embodiments. Figure 3 As shown, the method may include:

[0121] S301. Obtain the video stream to be detected.

[0122] It should be noted that the gesture recognition method provided in this application can be applied to a gesture recognition device or an electronic device integrating such a device. Here, the electronic device can be such as a computer, smartphone, tablet computer, laptop computer, PDA, personal digital assistant, navigation device, server, etc., and this application does not specifically limit its application. Furthermore, the electronic device used to execute the gesture recognition method and the electronic device used to execute the aforementioned model training method can be the same electronic device or different electronic devices, and this application also does not specifically limit their use.

[0123] It should also be noted that the video stream to be detected can be real-time acquired video data or non-real-time acquired video data. In some embodiments, acquiring the video stream to be detected may include:

[0124] Video data is acquired through the video acquisition module to determine the initial video stream; wherein the initial video stream includes at least one video frame;

[0125] Each video frame in the initial video stream is preprocessed to obtain the video stream to be detected.

[0126] It should be noted that, in order to obtain more accurate gesture recognition results, the embodiments of this application may first acquire an initial video stream, which includes at least one video frame; for example, real-time or non-real-time video data can be acquired by a video acquisition module in an electronic device to determine the initial video stream; then each video frame in the initial video stream is preprocessed to obtain a preprocessed video stream, which is the video stream to be detected.

[0127] Furthermore, in some embodiments, the preprocessing of each video frame in the initial video stream may include:

[0128] Resize each video frame in the initial video stream; and / or,

[0129] Perform color mode conversion on each video frame in the initial video stream.

[0130] It should be noted that video frame preprocessing can specifically include resizing and color mode conversion. For example, resizing the video frame to the same size as the sample images used to train the gesture recognition model, and converting the video frame's color mode to the same color mode as the sample images, helps the gesture recognition model to perform gesture recognition more quickly and accurately.

[0131] For example, the size of the video frame can be adjusted to a pixel size of 320×320, and the color mode of the video frame can be converted to RGB color mode (the inventors have proven through actual testing that image data in RGB color mode is more conducive to subsequent gesture recognition).

[0132] S302. Use a gesture recognition model to perform gesture detection on each video frame in the video stream to be detected, and determine the gesture type and hand positioning information in each video frame.

[0133] It should be noted that the gesture recognition model obtained through the aforementioned embodiments can be used to perform gesture detection on each video frame in the video stream to be detected, thereby obtaining the gesture recognition result for each video frame. The gesture recognition result can specifically include the gesture type and hand positioning information of each hand in each video frame.

[0134] In some embodiments, the hand positioning information may include the center coordinates of the hand positioning frame and the width and height of the hand positioning frame.

[0135] It should be noted that for any video frame (or image to be detected), see [reference needed]. Figure 4 This illustration shows a schematic diagram of a gesture recognition result for an image provided in an embodiment of this application. For example... Figure 4 As shown, the gesture recognition result for this image is: gesture type is victory-1.0. The white box represents the hand positioning bounding box. However, the center coordinates, width, and height of the hand positioning bounding box are not specified. Figure 4 As shown in the image.

[0136] In other words, for each video frame of the video stream to be detected, it is possible to obtain... Figure 4 The gesture recognition result shown is shown. However, if the video frame itself does not contain a hand, then naturally, a similar result cannot be obtained. Figure 4 The gesture recognition results shown do not identify the gesture type or hand positioning information.

[0137] Furthermore, in some embodiments, before performing IOU detection on the hand positioning information in the video stream to be detected, the method may further include:

[0138] Based on the hand positioning information in each video frame, determine the size of the hand positioning box in each video frame;

[0139] The size of the hand positioning frame in each video frame is compared with a preset size threshold.

[0140] Based on the comparison results, hands whose hand positioning box size is smaller than a preset size threshold in the video stream to be detected are filtered out, and the filtered video stream is determined as the video stream to be detected.

[0141] It should be noted that after determining the hand positioning information for each video frame, this embodiment further determines the size of the hand positioning box in each video frame accordingly, and filters out hands whose hand positioning box size is smaller than a preset size threshold, thereby eliminating the interference of an excessively small hand detection box on the accuracy of gesture recognition. For example, the preset size threshold is a pixel size of 50×50.

[0142] Specifically, the size of the hand positioning box in each video frame can be compared with a preset size threshold. Then, based on the comparison results, hands whose hand positioning box size is smaller than the preset size threshold in the video stream to be detected are filtered out. That is, hands whose hand positioning box size is greater than or equal to the preset size threshold are retained, while hands whose hand positioning box size is smaller than the preset threshold are filtered out. The filtered video stream is then determined as the video stream to be detected.

[0143] S303. Perform IOU detection on the hand positioning information in the video stream to be detected, and determine the hand detection result of the video stream to be detected.

[0144] It should be noted that, for the video stream to be detected, the hand detection result of the video stream is determined by performing Intersection Over Union (IOU) detection on the hand positioning information in the video stream. The hand detection result is used to indicate whether the hands in the video stream are the same hand.

[0145] In some embodiments, performing IOU detection on the hand location information in the video stream to be detected and determining the hand detection result of the video stream to be detected may include:

[0146] Identify at least one hand tracking group in the video stream to be detected; wherein each hand tracking group includes two video frame sequence numbers to be detected for IOU and the corresponding hand positioning information;

[0147] In each hand tracking group, if the difference in video frame sequence numbers in the hand tracking group is less than a preset difference threshold, and the IOU of hand positioning information in the hand tracking group is greater than a preset overlap threshold, then the hand data detected in the hand tracking group is determined to be the same hand.

[0148] If the hand data detected in at least one hand tracking group are all the same hand, the hand detection result indicates that the hand in the video stream to be detected is the same hand.

[0149] Furthermore, in some embodiments, the method may further include: if the difference in video frame sequence numbers in the hand tracking group is not less than a preset difference threshold, or if the IOU of hand positioning information in the hand tracking group is not greater than a preset overlap threshold, then the hand data detected in the hand tracking group is determined to be different hands.

[0150] It should be noted that there may be one or more hands in the video stream to be detected. Based on fully considering the temporal characteristics and taking advantage of the relatively slow movement of hands, this embodiment of the application uses a simple algorithm for Interchange of Units (IOU) to track one or more hands in the video stream to be detected. Specifically, at least one hand tracking group is determined from the video stream to be detected. Each hand tracking group includes two video frame sequence numbers to be detected and corresponding hand positioning information. The video frame sequence number is the Identity Document (ID) of each video frame in the video stream. For example, if a video stream to be detected includes ten video frames, then the video frame sequence numbers of each video frame can be 1, 2, 3...10 according to the time sequence. That is, a hand tracking group includes two hand data from two different video frames. IOU detection can determine whether these two hand data belong to the same hand.

[0151] For each hand tracking group, if the difference in video frame sequence numbers within that group is less than a preset difference threshold (e.g., 5), and the Intersection over Union (IOU) of the hand location information (primarily the IOU of the hand bounding box) is greater than a preset overlap threshold (e.g., 0.8), then the two hand data points in that hand tracking group are determined to be the same hand; otherwise, the two hand data points in that hand tracking group are determined to be different hands. Thus, when it is determined that the hand data detected in at least one hand tracking group are all the same hand, the hand detection result indicates that the hand in the video stream being detected is the same hand.

[0152] It should also be noted that currently, if there is a need for hand tracking, it is usually necessary to train an additional target tracking model (e.g., the MeanShift algorithm model), which increases the time and space redundancy of the algorithm and consumes more resources. However, the method of the embodiments in this application can perform hand tracking on at least one hand in the video stream to be detected, achieving hand tracking function without introducing a target tracking model, thereby reducing algorithm redundancy and saving resources.

[0153] S304. When the hand detection result indicates that the hand in the video stream to be detected is the same hand, determine the gesture type recognition result of the same hand in the video stream to be detected based on the gesture type in each video frame.

[0154] It should be noted that after identifying the same hand in the video stream to be detected, the gesture type recognition result of that same hand is determined based on the gesture type of each video frame in the video stream to be detected. Specifically, in some embodiments, determining the gesture type recognition result of the same hand in the video stream to be detected based on the gesture type in each video frame may include:

[0155] Obtain the gesture type of the same hand in the current video frame and the initial gesture type recognition results in a preset number of video frames before the current video frame;

[0156] The initial gesture type recognition result is optimized by using the gesture type in the current video frame to determine the gesture type recognition result of the same hand in the current video frame.

[0157] It should be noted that when determining the gesture type recognition result of the same hand in the video stream to be detected, the embodiments of this application still fully consider the temporal characteristics of gesture recognition, obtain the gesture recognition type of the hand in the current video frame and the initial gesture type recognition result in a preset number of video frames before the current video frame. The initial gesture type recognition result includes the gesture type of the hand in each of the preset number of video frames; for example, the preset number can be five frames.

[0158] Then, the initial gesture type recognition result is optimized to determine the gesture type recognition result of the same hand in the current video frame, until the last frame of the video stream to be detected is recognized.

[0159] Furthermore, in some embodiments, optimizing the initial gesture type recognition result using the gesture type in the current video frame to determine the gesture type recognition result of the same hand in the current video frame may include:

[0160] Based on the gesture type in the current video frame and the initial gesture type recognition results in a preset number of video frames before the current video frame, a gesture type candidate set is formed;

[0161] The most frequently occurring gesture type is determined from the candidate set of gesture types, and the most frequently occurring gesture type is identified as the gesture type recognition result of the same hand in the current video frame.

[0162] It should be noted that the gesture type candidate set consists of the gesture types of the same hand in the current video frame and the gesture types in a preset number of video frames preceding the current video frame. The gesture type that appears most frequently is determined as the final gesture type recognition result for that same hand. This also eliminates the possibility of recognition errors due to errors in the gesture recognition model or noise.

[0163] This embodiment provides a gesture recognition method, which involves acquiring a video stream to be detected; using a gesture recognition model to perform gesture detection on each video frame in the video stream to determine the gesture type and hand location information in each video frame; performing IOU detection on the hand location information in the video stream to determine the hand detection result in the video stream to be detected; and when the hand detection result indicates that the hand in the video stream to be detected is the same hand, determining the gesture type recognition result of the same hand in the video stream to be detected based on the gesture type in each video frame. In this way, by performing gesture detection on each video frame in the video stream to be detected, and tracking the hands in the video stream based on the detected hand positioning information, the same hand in the video stream to be detected is identified, and the gesture type recognition result of the same hand in the video stream to be detected is further determined based on the detected gesture type; thus improving the accuracy and precision of gesture recognition. In addition, since the video frame sequence number and hand positioning information are combined when determining whether the hand data in the video stream to be detected belong to the same hand, gesture recognition of multiple hands can be performed separately without interference, based on full consideration of temporal features, without the need to introduce other target tracking models, thus optimizing the efficiency of gesture recognition using the algorithm, effectively reducing the spatial and temporal redundancy of the algorithm, and saving computing resources. Furthermore, when determining the gesture type recognition result of the same hand, the temporal features between video frames are also fully considered, improving the stability of gesture type recognition and further improving the accuracy of gesture type recognition, while reducing the impact of model errors and noise on the recognition results.

[0164] In yet another embodiment of this application, Figure 5 A schematic diagram of the architecture of a gesture recognition device provided in an embodiment of this application is shown. Figure 5 As shown, the architecture may include a real-time video acquisition module, a data processing module, a gesture recognition module, a hand tracking module, and an optimization and correction module; thus, the gesture recognition method of this application embodiment can be completed through the collaboration between the modules.

[0165] based on Figure 5 The architecture shown is described in the following document. Figure 6 The diagram illustrates a detailed flowchart of a gesture recognition method provided in an embodiment of this application. Figure 6 As shown, the detailed process may include:

[0166] S601, Obtain video stream.

[0167] It should be noted that, firstly, a real-time video stream or a non-real-time video stream (equivalent to the initial video stream in the aforementioned embodiments) can be acquired through a real-time video acquisition module. This application embodiment does not specifically limit this.

[0168] S602, Obtain video frames.

[0169] S603. Preprocess the video frames.

[0170] It should be noted that after acquiring the video stream, the data processing module can extract video frames from the video stream and preprocess each extracted video frame. Specifically, preprocessing video frames can include resizing images, for example, adjusting the image size to 320×320 pixels; preprocessing video frames can also include converting the color mode of the video frames, for example, converting the video frames to a red-green-blue color mode.

[0171] S604. Perform gesture information detection and filtering on the preprocessed video frames.

[0172] It should be noted that for the preprocessed video frames, a gesture recognition module can be used to identify gesture information in each frame using a gesture recognition model, and then perform filtering. The gesture recognition results output by the model can include the coordinates of the center point of the hand bounding box, the width and height of the bounding box, the gesture type, and its corresponding confidence score. The recognition results can be referenced as described above. Figure 4 .

[0173] It should also be noted that the filtering process may include filtering hand frames whose size (area) is smaller than a preset size (area) threshold γ1, for example, filtering hands with a preset size of 50 pixels × 50 pixels, to eliminate the interference of an excessively small hand frame on gesture recognition accuracy. In this way, the video frames that have undergone preprocessing and filtering constitute the video stream to be detected.

[0174] S605. Track the multiple hands in the video stream to be detected separately based on the hand positioning information and video frame sequence number.

[0175] It should be noted that a hand tracking module can be used to track multiple hands separately in the video stream to be detected, based on hand positioning information and video frame sequence numbers. The hand positioning information can include the coordinates of the center point of the aforementioned hand positioning box, the width and height of the hand positioning box, etc., combined with the video frame sequence numbers, thus enabling the tracking of multiple hands separately while fully considering temporal characteristics.

[0176] Specifically, if the difference between the video frame numbers of two different video frames is less than a preset difference threshold γ2 (e.g., 5) and the Intersection over Union (IOU) of the hand localization boxes detected in these two video frames is greater than a preset overlap threshold γ3 (e.g., 0.80), then the hand data detected in these different video frames are considered to be the same hand; otherwise, the hand data detected in these different video frames are considered to be different hands. This hand tracking method can achieve hand tracking functionality without introducing a target tracking model. The video frame numbers of the two different video frames and the hand localization information corresponding to the hands from the two different video frames constitute the hand tracking group described in the aforementioned embodiment.

[0177] S606. Determine the final gesture type recognition result.

[0178] It should be noted that the final gesture type recognition result can be determined by optimizing the correction module, taking into account the temporal features of gesture recognition. Specifically, for the same hand, its position in the current frame t is obtained. i and the first n frames (t) i-1 t i-2 ... t i-n For example, the preliminary gesture type recognition results in video frames n=5 (i.e., the gesture type in each of the first n frames) are used to determine the gesture type with the highest frequency in the current video frame t. i The second gesture type, which is the final gesture type recognition result.

[0179] S607, Has the last frame been detected?

[0180] It should be noted that if the judgment result is yes, then the last frame of the video stream to be detected has been identified, and the final gesture type recognition result can be directly output, and the process ends. If the judgment result is no, then the last frame of the video stream to be detected has not been identified, and the process can return to step S602 until the last frame is identified. For example, when a user interacts with an electronic device through gestures, the electronic device can respond or perform operations according to the recognized gesture type recognition result.

[0181] In other words, the gesture recognition method provided in this application embodiment can be a real-time static gesture recognition method. Real-time performance is reflected in its ability to perform gesture recognition on real-time captured video streams, while static performance is reflected in the gesture detection of each video frame in the video stream using a gesture recognition model. On one hand, this method selects six feature layers from a preset network model to extract features from images, reducing the algorithm complexity of the model and achieving real-time output of recognition results. From video stream acquisition to gesture recognition output, the CPU time is only about 40ms, enabling mobile applications. On the other hand, this method fully considers the temporal characteristics of video frames, improving gesture recognition accuracy. Furthermore, based on the relatively slow nature of hand movements, this method implements hand tracking functionality through a simple IOU discrimination algorithm, eliminating the need for a target tracking model, optimizing algorithm efficiency, and effectively reducing spatial and temporal redundancy. It achieves multi-hand gesture recognition functionality while fully considering temporal characteristics. Finally, this method performs image fusion processing on the acquired gestures, increasing the diversity of the sample image set and effectively improving the robustness of the gesture recognition model. Therefore, this application embodiment provides a real-time, multi-hand static gesture recognition method based on temporal features.

[0182] In addition, the embodiments of this application also include the acquisition and preprocessing of the training set (i.e., the sample image set in the aforementioned embodiments) for the gesture recognition model, and the training of the gesture recognition model. Specifically, the acquisition and preprocessing of the training set for the gesture recognition model may include: acquiring raw gesture image data (i.e., at least one gesture image in the aforementioned embodiments) in a real-world scenario, performing image fusion processing on the gesture images using image fusion technology, and annotating hand information (i.e., adding annotation information in the aforementioned embodiments).

[0183] Image fusion technology refers to the technique of fusing captured gesture images with other types of image data (i.e., non-gesture images in the aforementioned embodiments, such as landscape images), including but not limited to fusion and stitching techniques. The process of image fusion processing for gesture images can be found in [link to relevant documentation]. Figure 2 It merges a hand gesture image with a landscape image.

[0184] In addition, annotation information can be added before or after image fusion processing. The annotation tool used when adding annotation information can be a publicly known annotation tool in the field, such as the annotation tool VGG ImageAnnotator. The annotation tool can be used to annotate hand information such as hand positioning boxes.

[0185] It should also be noted that, in this embodiment, when the gesture recognition model performs the target detection task (in this embodiment, the target detection task includes a type classification task for detecting gesture types and a localization regression task for determining hand positioning information), it needs to simultaneously focus on the classification accuracy of the gesture type and the regression accuracy of the hand positioning information. Therefore, it needs to pay attention to both the semantic information and contour detail information of the feature layer. Existing network structures (for example, see...) Figure 7 (This illustration shows a hierarchical fusion diagram of a network structure provided in an embodiment of this application.) Often, it only focuses on the fusion of high-level semantic features to low-level and mid-level contour detail features or the fusion of low-level and mid-level contour detail features to high-level semantic features, and adopts a layer-by-layer cascade / sequential fusion method, which cannot achieve a good recognition effect.

[0186] Based on this, the embodiments of this application use an improved version of SSD_mobilenetV3_I as the preset network model to realize the interactive fusion of low-level and mid-level contour detail features and high-level semantic features, and adopt a parallel and independent feature fusion method so that each feature layer has both high-level semantic features and low-level and mid-level contour detail features.

[0187] See Figure 8 This illustrates a schematic diagram of the composition structure of a preset network model provided in an embodiment of this application. For example... Figure 8As shown, the preset network model can include a feature extraction network, a classification branch network, and a regression branch network. The classification branch network and regression branch network can be collectively referred to as the detection head or detection prediction layer. The feature extraction network can include a feature extraction layer, a feature fusion layer, and a feature convolutional layer. Each feature convolutional layer includes two Conv1×1 layers. When a sample image is input into the preset network model, the feature extraction network first performs feature interaction fusion on the six layers (Layer 14, Layer 17, Layer 20, Layer 23, Layer 26, and Layer 29) of the backbone network SSD_mobilenetV3 through operations such as upsampling, downsampling, concatenation, and convolution (Conv(filter=(1,1)), Conv1×1). The fused six-layer features are then input into the classification branch network and regression branch network respectively to determine the gesture type and hand positioning information. Upsampling of each feature layer can be performed using the Deconvolution function. Specifically, the Deconvolution function parameters for Layer 17, Layer 20, and Layer 29 can be set to kernel_size = [1,2,2,1], stride = [1,2,2,1], and padding = 0; while the Deconvolution function parameters for Layer 23 and Layer 26 can be set to kernel_size = [1,2,2,1], stride = [1,2,2,1], padding = 1, and output_padding = 1. Downsampling of each feature layer can be performed using the max_pooling function. The max_pooling function parameters for Layer14, Layer17, and Layer26 can be set to kernel_size = [1,2,2,1], stride = [1,2,2,1], and padding = 0. The max_pooling function parameters for Layer20 and Layer23 can be set to kernel_size = [1,2,2,1], stride = [1,2,2,1], and padding = 1.

[0188] After completing upsampling and downsampling respectively, as Figure 8As shown, each feature layer is then fused through a concatenation operation and two convolution operations (with convolutional layer parameters kernel_size = [1,1,1,1], stride = [1,1,1,1], padding = 0), resulting in the fused target feature maps for each of the six feature layers. Here, kernel_size represents the convolutional kernel size, stride represents the convolutional stride, padding represents the feature map padding width, and output_padding represents the output feature edge augmentation value.

[0189] In this way, by performing feature interaction fusion on the feature maps of some feature layers in the preset network model, not only are the features of the target feature map of each layer enriched, but also the complexity of the algorithm can be reduced by selecting only some feature layers for feature interaction fusion, so as to achieve real-time output of gesture recognition results.

[0190] Furthermore, the inventors demonstrated through extensive experimental results that when performing object detection tasks within a pre-defined network model, the features focused on by the type classification task and the hand localization information regression task differ; the former focuses more on semantic features, while the latter focuses more on contour detail features. Based on this, the embodiments of this application further improve the detection head of the pre-defined network model (the detection head includes a classification branch network and a regression branch network) by introducing attention modules into both the classification branch network and the regression branch network.

[0191] See Figure 9 This illustrates a schematic diagram of the composition structure of a classification branch network and a regression branch network provided in an embodiment of this application. Figure 9 As shown, each fused target feature map is input into the attention modules of the classification and regression branches respectively. For each target feature map, the attention module extracts features through convolutions Conv 3×3 (kernel_size=[1,3,3,1], stride=[1,2,2,1], padding=1), Conv 1×1 (kernel_size=[1,1,1,1], stride=[1,1,1,1], padding=0) and transposed convolutions Deconv 3×3 (kernel_size=[1,3,3,1], stride=[1,2,2,1], padding=1, output_padding=1), and performs skip connection processing to capture information of different proportions.

[0192] Taking the classification branch network as an example, for any layer's target feature map, it first undergoes a series of convolutional processes to obtain the features of the target feature map after the classification branch convolution (i.e., the initial classification feature matrix in the aforementioned embodiment). Then, the convolutional features are processed by an activation function (softmax function) to select attention features, obtaining the probability value matrix of each pixel (i.e., the classification probability matrix in the aforementioned embodiment). Then, the probability value matrix of each point is multiplied element-wise with the target feature map, which "amplifies" important features (important features have larger probability values). The result of the multiplication (i.e., the classification product matrix in the aforementioned embodiment) is then added element-wise with the target feature map to obtain the classification sum matrix. This method is used to process the target feature map of each layer, thereby obtaining the classification sum matrices corresponding to the six feature layers. Then, convolution is performed for deduplication, etc., to predict the gesture type until a first sub-model with high accuracy is obtained.

[0193] In this embodiment, the attention module can employ a mixed attention mechanism, selecting attention features in both the channel and spatial dimensions. The mixed attention mechanism can be implemented using the sigmoid function, meaning the activation function is the sigmoid function. This involves processing each pixel in both the channel and spatial dimensions using the sigmoid function. The sigmoid function (also called the Logistic function) is one of the important activation functions in neural networks, as shown in formula (1).

[0194]

[0195] Where i is the pixel spatial position, c is the pixel channel position, and x i,c This is pixel data.

[0196] In summary, the key points of the embodiments of this application are mainly as follows:

[0197] (1) The process of forming the sample image set for the gesture recognition model includes: collecting gesture images in real-world scenes; and performing image fusion processing on the gesture images and non-gesture images (such as landscapes), including but not limited to fusion and stitching techniques. The sample image set formed in this way enables the final gesture recognition model to be adaptable to real-world scenes, possess strong robustness, and reduce interference from external factors such as lighting.

[0198] (2) The preset network model used in this application embodiment is the improved SSD_mobilenetV3_I model. It achieves the interaction and fusion of low-level and mid-level contour detail features and high-level semantic features through operations such as upsample, downsample, concatenate, and Conv1×1, and adopts a parallel and independent feature fusion method. In addition, this application embodiment also improves the detection head of the preset network model, that is, it introduces mixed attention modules into the classification branch network and the regression branch network respectively.

[0199] (3) In the embodiment of this application, when performing gesture recognition, the hand tracking function is realized by the hand positioning box (IOU) detected in the preceding and following video frames. Under the premise of fully considering the temporal characteristics of the preceding and following video frames and without introducing a target tracking model, multiple hand gestures can be recognized without interference.

[0200] In short, the gesture recognition method proposed in this application can be implemented through the following modules and steps:

[0201] S1, Real-time video acquisition module, which acquires real-time video data, or non-real-time video data.

[0202] S2, the data processing module, extracts video frames and performs preprocessing, including adjusting the image size (320×320) and converting the color mode to RGB mode (data tests show that RGB format image data is more beneficial to the subsequent hand recognition module).

[0203] S3, the gesture recognition module, uses a gesture recognition model to identify and filter gesture information in the preprocessed video frames. The gesture information output by the gesture recognition model includes the coordinates of the center point of the hand positioning frame, the width and height of the hand positioning frame, the gesture recognition type, and its corresponding confidence level. The filtering operation involves filtering out hands whose hand positioning frame area is smaller than a preset size threshold to eliminate interference from excessively small hand positioning frames on gesture recognition accuracy.

[0204] S4, the hand tracking module, tracks multiple hands separately based on hand positioning information and video frame IDs (video frame sequence numbers), thereby achieving gesture recognition for multiple hands while fully considering temporal characteristics. If the difference between video frame IDs is less than a preset difference threshold and the Intersection over Union (IOU) of the hand locations detected in these two video frames is greater than a preset overlap threshold, then the hand data detected in these different video frames are considered as the same hand data; otherwise, the hand data detected in these different video frames are considered as different hand data. This hand tracking method achieves hand tracking functionality without introducing a target tracking model.

[0205] S5. The optimization and correction module considers the temporal characteristics of gesture recognition and determines the final gesture recognition result. For the same hand data, it obtains the preliminary gesture recognition results in the current frame ti and the previous n frames of video data, and takes the gesture type with the highest frequency as the final gesture recognition result for the current video frame ti.

[0206] This application also includes the acquisition and preprocessing of a sample image set for the gesture recognition model, and the training of the gesture recognition model. The acquisition and preprocessing of the sample image set for the gesture recognition model includes: acquiring gesture images in real-world scenes, performing image fusion processing on the gesture images, and annotating hand information. Image fusion involves fusing the acquired gesture images with other types of image data (such as landscapes), including but not limited to fusion and stitching.

[0207] This application's embodiment employs an improved version of the SSD_mobilenetV3_I model to achieve interactive fusion of low-level and mid-level contour detail features with high-level semantic features. It also adopts a parallel and independent feature fusion method, ensuring that each feature layer simultaneously possesses high-level semantic features and low-level and mid-level contour detail features. Its overall structure is as follows: Figure 8 As shown in the diagram, this structure uses upsampling, downsampling, concatenation, and convolution operations to perform feature interaction fusion on the six layers (layer 14, layer 17, layer 20, layer 23, layer 26, and layer 29) of the backbone network SSD_mobilenetV3. The fused six-layer features (target feature maps) are then input into the detection head for gesture type determination and hand localization information regression.

[0208] This application embodiment also improves the detection head of the preset network model by introducing attention modules into the classification branch network and the regression branch network respectively. This attention module extracts features from the target feature map through convolution and performs skip connections to capture information at different proportions, as shown in the following structure: Figure 9 As shown, this attention module employs a hybrid attention mechanism, selecting attention features in both the channel and spatial dimensions. The hybrid attention mechanism is implemented using the sigmoid function, which processes each pixel in both the channel and spatial dimensions using the sigmoid function.

[0209] This application provides a gesture recognition method. The specific implementation of the aforementioned embodiments is described in detail through the above embodiments. It can be seen from the above embodiments that, compared with related technologies, the gesture recognition method provided by this application has at least the following advantages: (1) Real-time performance. After a large number of experimental verifications, using the gesture recognition method provided by this application, from video stream acquisition to gesture recognition result output, the CPU time is about 40ms (operating system Windows 10, memory 16G, system type 64-bit, processor Core(TM) i7-8700). Under the same conditions, the gesture recognition method based on YOLO network provided by related technologies requires 300+ms to detect a single image. It can be seen that the method provided by this application greatly improves the efficiency of gesture recognition, has stronger real-time performance, and can realize mobile terminal applications. (2) High model accuracy. The gesture recognition model obtained by training the improved SSD_mobilenetV3_I model has significantly improved the accuracy of gesture type classification and hand positioning information regression. (3) By optimizing the correction module, the temporal characteristics between video frames are fully considered, which improves the stability of gesture recognition and further improves the accuracy of gesture recognition. (4) By using the hand positioning information and video frame sequence number detected between different video frames, the function of recognizing gestures of multiple hands separately without interference is realized on the basis of fully considering the temporal characteristics. The implementation of this function does not introduce MeanShift or other target tracking algorithm models, optimizes the algorithm efficiency, effectively reduces the spatial and temporal redundancy of the algorithm, and saves computing resources; (5) By performing image fusion processing on gesture images, the robustness of the gesture recognition model is improved.

[0210] In another embodiment of this application, see [link to application]. Figure 10 This illustrates a schematic diagram of the composition of a model training device 100 provided in an embodiment of this application. For example... Figure 10 As shown, the model training device 100 may include a first acquisition unit 1001, a fusion unit 1002, and a training unit 1003, wherein,

[0211] The first acquisition unit 1001 is configured to acquire at least one gesture image and at least one non-gesture image;

[0212] The fusion unit 1002 is configured to perform image fusion processing on each gesture image in the at least one gesture image and the at least one non-gesture image to obtain a sample image set;

[0213] Training unit 1003 is configured to train a preset network model using the sample image set to obtain a gesture recognition model; wherein, the gesture recognition model includes a first sub-model and a second sub-model, the first sub-model is used to determine the gesture type of the image to be detected, and the second sub-model is used to determine the hand positioning information of the image to be detected.

[0214] In some embodiments, the image fusion process includes at least: image fusion processing and / or image stitching processing.

[0215] In some embodiments, the first acquisition unit 1001 is further configured to add annotation information to the sample images in the sample image set; wherein the annotation information includes gesture type information in the sample image and hand positioning information in the sample image, and the hand positioning information includes the center coordinates of the hand positioning frame and the width and height of the hand positioning frame.

[0216] In some embodiments, the preset network model includes a feature extraction network, a classification branch network, and a regression branch network; the training unit 1003 is further configured to input sample images from the sample image set into the preset network model; and to perform feature processing on the sample images from the sample image set through the feature extraction network to obtain at least one target feature map; and to train the classification branch network through the at least one target feature map to obtain a first sub-model; and to train the regression branch network through the at least one target feature map to obtain a second sub-model; and to determine the gesture recognition model based on the first sub-model and the second sub-model.

[0217] In some embodiments, the feature extraction network includes a feature extraction layer, a feature fusion layer, and a feature convolutional layer; the training unit 1003 is further configured to use the feature extraction layer to perform initial feature extraction on the sample image to determine at least one initial feature map corresponding to the sample image; and use the feature fusion layer to perform feature interaction fusion on the at least one initial feature map to obtain at least one fused feature map corresponding to the sample image; and use the feature convolutional layer to perform a convolution operation on the at least one fused feature map to obtain the at least one target feature map.

[0218] In some embodiments, the training unit 1003 is further configured to: determine the initial feature map of the sample image in the i-th feature layer and the initial feature maps of other feature layers besides the i-th feature layer; and perform sampling processing on the initial feature maps of other feature layers besides the i-th feature layer to obtain a sampling result; and use a feature fusion layer to perform feature fusion on the initial feature map of the i-th feature layer and the sampling result to obtain a fused feature map of the sample image in the i-th feature layer; wherein i is an integer greater than zero.

[0219] In some embodiments, the feature layers other than the i-th feature layer include: a first feature layer portion located before the i-th feature layer and a second feature layer portion located after the i-th feature layer, wherein the level of the first feature layer portion is higher than the level of the i-th feature layer, and the level of the i-th feature layer is higher than the level of the second feature layer portion; the training unit 1003 is further configured to perform downsampling processing on the initial feature map of the first feature layer portion and upsampling processing on the initial feature map of the second feature layer portion to obtain the sampling result.

[0220] In some embodiments, the downsampling process employs a max pooling function, and the upsampling process employs a deconvolution function.

[0221] In some embodiments, the preset network model includes the SSD_mobilenetV3_I model; wherein, the feature extraction layer includes feature layers Layer14, Layer17, Layer20, Layer23, Layer26 and Layer29 in the SSD_mobilenetV3_I model, the feature fusion layer includes a cascaded concatenate layer, and the feature convolutional layer includes a convolutional layer with a kernel size of 1×1.

[0222] In some embodiments, the classification branch network includes an attention mechanism module; the training unit 1003 is further configured to determine a first model loss function, perform classification training on the at least one target feature map using the classification branch network including the attention mechanism module, and determine the trained classification branch network as the first sub-model when the value of the first model loss function reaches a preset convergence value; wherein, the attention mechanism module includes a convolution sub-module, an activation sub-module, a multiplication sub-module, and an addition sub-module, and the convolution sub-module consists of a convolutional layer, a transposed convolutional layer, and a skip layer.

[0223] In some embodiments, the regression branch network includes an attention mechanism module; the training unit 1003 is further configured to determine a second model loss function, perform regression training on the at least one target feature map using the regression branch network including the attention mechanism module, and determine the trained regression branch network as the second sub-model when the value of the second model loss function reaches a preset convergence value; wherein, the attention mechanism module includes a convolution sub-module, an activation sub-module, a multiplication sub-module, and an addition sub-module, and the convolution sub-module consists of a convolutional layer, a transposed convolutional layer, and a skip layer.

[0224] In another embodiment of this application, see [reference needed]. Figure 11This illustration shows a schematic diagram of the composition structure of a gesture recognition device 110 provided in an embodiment of this application. Figure 11 As shown, the gesture recognition device 110 may include a second acquisition unit 1101, a gesture detection unit 1102, an IOU detection unit 1103, and a determination unit 1104, wherein,

[0225] The second acquisition unit 1101 is configured to acquire the video stream to be detected;

[0226] The gesture detection unit 1102 is configured to use the gesture recognition model to perform gesture detection on each video frame in the video stream to be detected, and to determine the gesture type and hand positioning information in each video frame;

[0227] IOU detection unit 1103 is configured to perform overlap IOU detection on the hand positioning information in the video stream to be detected, and determine the hand detection result of the video stream to be detected;

[0228] The determining unit 1104 is configured to determine the gesture type recognition result of the same hand in the video stream to be detected based on the gesture type in each video frame when the hand detection result indicates that the hand in the video stream to be detected is the same hand.

[0229] In some embodiments, the second acquisition unit 1101 is further configured to acquire video data through a video acquisition module to determine an initial video stream; wherein the initial video stream includes at least one video frame; and to preprocess each video frame in the initial video stream to obtain the video stream to be detected.

[0230] In some embodiments, the second acquisition unit 1101 is further configured to resize each video frame in the initial video stream; and / or to convert the color mode of each video frame in the initial video stream.

[0231] In some embodiments, the hand positioning information includes the center coordinates of the hand positioning frame and the width and height of the hand positioning frame.

[0232] In some embodiments, the second acquisition unit 1101 is further configured to determine the size of the hand positioning frame in each video frame based on the hand positioning information in each video frame; compare the size of the hand positioning frame in each video frame with a preset size threshold; and filter out hands in the video stream to be detected whose hand positioning frame size is smaller than the preset size threshold according to the comparison result, and determine the filtered video stream as the video stream to be detected.

[0233] In some embodiments, the IOU detection unit 1103 is further configured to determine at least one hand tracking group in the video stream to be detected; wherein each hand tracking group includes two video frame sequence numbers to be detected for IOU and corresponding hand positioning information; and in each hand tracking group, if the difference between the video frame sequence numbers in the hand tracking group is less than a preset difference threshold, and the IOU of the hand positioning information in the hand tracking group is greater than a preset overlap threshold, then the hand data detected in the hand tracking group is determined to be the same hand; and if the hand data detected in at least one hand tracking group are all the same hand, then the hand detection result indicates that the hand in the video stream to be detected is the same hand.

[0234] In some embodiments, the IOU detection unit 1103 is further configured to determine that the hand data detected in the hand tracking group are different hands if the difference in the video frame sequence number in the hand tracking group is not less than a preset difference threshold, or if the IOU of the hand positioning information in the hand tracking group is not greater than a preset overlap threshold.

[0235] In some embodiments, the determining unit 1104 is further configured to acquire the gesture type of the same hand in the current video frame and the initial gesture type recognition result in a preset number of video frames before the current video frame; and to optimize the initial gesture type recognition result using the gesture type in the current video frame to determine the gesture type recognition result of the same hand in the current video frame.

[0236] In some embodiments, the determining unit 1104 is further configured to form a gesture type candidate set based on the gesture type in the current video frame and the initial gesture type recognition results in a preset number of video frames prior to the current video frame; and to determine the gesture type with the highest frequency of occurrence from the gesture type candidate set, and to determine the gesture type with the highest frequency of occurrence as the gesture type recognition result of the same hand in the current video frame.

[0237] Understandably, in this embodiment, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and can also be a module or a non-modular component. Furthermore, the components in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0238] If the integrated unit is implemented as a software functional module and not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0239] Therefore, this embodiment provides a computer storage medium storing a computer program, which, when executed by a processor, implements any of the model training methods described in the foregoing embodiments, or implements any of the gesture recognition methods described in the foregoing embodiments.

[0240] Based on the above-described composition of a model training device 100 and / or gesture recognition device 40, and computer storage medium, see [link to relevant documentation]. Figure 12 This illustrates a schematic diagram of the specific hardware structure of an electronic device 120 provided in an embodiment of this application. For example... Figure 12 As shown, it may include: a communication interface 1201, a memory 1202, and a processor 1203; the various components are coupled together through a bus system 1204. It is understood that the bus system 1204 is used to implement communication between these components. In addition to a data bus, the bus system 1204 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 12 All buses are designated as bus system 1204. Among them, communication interface 1201 is used for receiving and transmitting signals during information exchange with other external network elements.

[0241] The memory 1202 is used to store computer programs that can run on the processor 1203;

[0242] Processor 1203, when running the computer program, performs the following:

[0243] Obtain at least one gesture image and at least one non-gesture image;

[0244] Each gesture image in the at least one gesture image is fused with the at least one non-gesture image to obtain a sample image set;

[0245] The preset network model is trained using the sample image set to obtain a gesture recognition model; wherein, the gesture recognition model includes a first sub-model and a second sub-model, the first sub-model is used to determine the gesture type of the image to be detected, and the second sub-model is used to determine the hand positioning information of the image to be detected;

[0246] Alternatively, processor 1203, when running the computer program, is configured to perform:

[0247] Obtain the video stream to be detected;

[0248] The gesture recognition model is used to perform gesture detection on each video frame in the video stream to be detected, and the gesture type and hand positioning information in each video frame are determined.

[0249] The hand location information in the video stream to be detected is subjected to overlap (IOU) detection to determine the hand detection result of the video stream to be detected;

[0250] When the hand detection result indicates that the hand in the video stream to be detected is the same hand, the gesture type recognition result of the same hand in the video stream to be detected is determined according to the gesture type in each video frame.

[0251] It is understood that the memory 1202 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 1202 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0252] The processor 1203 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 1203 or by instructions in software form. The processor 1203 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 1202. Processor 1203 reads the information in memory 1202 and completes the steps of the above method in conjunction with its hardware.

[0253] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0254] For software implementation, the techniques described herein can be achieved through modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented within the processor or externally.

[0255] Alternatively, as another embodiment, the processor 1203 is also configured to perform the method described in any of the foregoing embodiments when running the computer program.

[0256] In this embodiment, for the electronic device 120, during model training, since the sample images include not only gestures but also non-gesture content, the trained gesture recognition model is more suitable for real-world scenarios, enhancing the recognition accuracy and robustness of the gesture recognition model and reducing interference from external factors. During gesture recognition, by performing gesture detection on each video frame of the video stream to be detected, and tracking the hand in the video stream based on the detected hand positioning information, the same hand in the video stream is identified, and further, the gesture type recognition result of the same hand in the video stream is determined based on the detected gesture type; thereby improving the accuracy and precision of gesture recognition.

[0257] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

[0258] It should be noted that, in this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0259] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0260] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0261] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0262] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0263] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A gesture recognition method, characterized in that, include: Obtain the video stream to be detected; A gesture recognition model is used to detect gestures in each video frame of the video stream to be detected, and the gesture type and hand positioning information in each video frame are determined. The hand location information in the video stream to be detected is subjected to overlap (IOU) detection to determine the hand detection result of the video stream to be detected; When the hand detection result indicates that the hand in the video stream to be detected is the same hand, the gesture type recognition result of the same hand in the video stream to be detected is determined according to the gesture type in each video frame; The step of performing IOU detection on the hand positioning information in the video stream to be detected and determining the hand detection result of the video stream to be detected includes: determining at least one hand tracking group in the video stream to be detected; wherein each hand tracking group includes two video frame sequence numbers to be detected for IOU and corresponding hand positioning information; in each hand tracking group, if the difference between the video frame sequence numbers in the hand tracking group is less than a preset difference threshold, and the IOU of the hand positioning information in the hand tracking group is greater than a preset overlap threshold, then it is determined that the hand data detected in the hand tracking group is the same hand; if the hand data detected in at least one hand tracking group is the same hand, it is determined that the hand detection result indicates that the hand in the video stream to be detected is the same hand.

2. The method according to claim 1, characterized in that, The method further includes: Obtain at least one gesture image and at least one non-gesture image; Each gesture image in the at least one gesture image is fused with the at least one non-gesture image to obtain a sample image set; The gesture recognition model is obtained by training a preset network model using the sample image set; wherein the gesture recognition model includes a first sub-model and a second sub-model, the first sub-model is used to determine the gesture type of the image to be detected, and the second sub-model is used to determine the hand positioning information of the image to be detected.

3. The method according to claim 2, characterized in that, The image fusion process includes at least: image fusion processing and / or image stitching processing.

4. The method according to claim 2, characterized in that, The method further includes: Add annotation information to the sample images in the sample image set; The annotation information includes gesture type information and hand positioning information in the sample image. The hand positioning information includes the center coordinates of the hand positioning frame and the width and height of the hand positioning frame.

5. The method according to claim 2, characterized in that, The preset network model includes a feature extraction network, a classification branch network, and a regression branch network; The step of training a preset network model using the sample image set to obtain a gesture recognition model includes: Input the sample images from the sample image set into the preset network model; The feature extraction network is used to process the sample images in the sample image set to obtain at least one target feature map. The classification branch network is trained using the at least one target feature map to obtain the first sub-model; and the regression branch network is trained using the at least one target feature map to obtain the second sub-model. The gesture recognition model is determined based on the first sub-model and the second sub-model.

6. The method according to claim 5, characterized in that, The feature extraction network includes a feature extraction layer, a feature fusion layer, and a feature convolutional layer; the step of performing feature processing on sample images in the sample image set through the feature extraction network to obtain at least one target feature map includes: The feature extraction layer is used to perform initial feature extraction on the sample image to determine at least one initial feature map corresponding to the sample image. The feature fusion layer is used to perform feature interaction fusion on the at least one initial feature map to obtain at least one fused feature map corresponding to the sample image; The feature convolutional layer is used to perform a convolution operation on the at least one fused feature map to obtain the at least one target feature map.

7. The method according to claim 6, characterized in that, The step of using the feature fusion layer to perform feature interaction fusion on the at least one initial feature map to obtain at least one fused feature map corresponding to the sample image includes: Determine the initial feature map of the sample image in the i-th feature layer and the initial feature maps of other feature layers besides the i-th feature layer; The initial feature maps of the feature layers other than the i-th feature layer are sampled to obtain the sampling results; The initial feature map of the i-th feature layer and the sampling result are fused using a feature fusion layer to obtain the fused feature map of the sample image in the i-th feature layer; where i is an integer greater than zero.

8. The method according to claim 7, characterized in that, The other feature layers besides the i-th feature layer include: a first feature layer portion located before the i-th feature layer and a second feature layer portion located after the i-th feature layer, wherein the first feature layer portion has a higher level than the i-th feature layer, and the i-th feature layer has a higher level than the second feature layer portion. Accordingly, the sampling process for the initial feature maps of the feature layers other than the i-th feature layer to obtain the sampling results includes: The initial feature map of the first feature layer is downsampled and the initial feature map of the second feature layer is upsampled to obtain the sampling result.

9. The method according to claim 8, characterized in that, The downsampling process uses a max pooling function, and the upsampling process uses a deconvolution function.

10. The method according to claim 6, characterized in that, The preset network model includes the SSD_mobilenetV3_I model; The feature extraction layer includes the feature layers Layer14, Layer17, Layer20, Layer23, Layer26 and Layer29 in the SSD_mobilenetV3_I model; the feature fusion layer includes a cascaded concatenation layer; and the feature convolutional layer includes a convolutional layer with a kernel size of 1×1.

11. The method according to claim 5, characterized in that, The classification branch network includes an attention mechanism module; training the classification branch network using the at least one target feature map to obtain the first sub-model includes: A first model loss function is determined, and the classification branch network including the attention mechanism module is used to classify and train the at least one target feature map. When the value of the first model loss function reaches a preset convergence value, the trained classification branch network is determined as the first sub-model. The attention mechanism module includes a convolution submodule, an activation submodule, a multiplication submodule, and an addition submodule. The convolution submodule consists of a convolutional layer, a transposed convolutional layer, and a skip layer.

12. The method according to claim 5, characterized in that, The regression branch network includes an attention mechanism module; the step of training the regression branch network using the at least one target feature map to obtain the second sub-model includes: Determine the second model loss function, use the regression branch network including the attention mechanism module to perform regression training on the at least one target feature map, and when the value of the second model loss function reaches a preset convergence value, determine the trained regression branch network as the second sub-model; The attention mechanism module includes a convolution submodule, an activation submodule, a multiplication submodule, and an addition submodule. The convolution submodule consists of a convolutional layer, a transposed convolutional layer, and a skip layer.

13. The method according to any one of claims 1-12, characterized in that, The acquisition of the video stream to be detected includes: Video data is acquired through a video acquisition module to determine an initial video stream; wherein the initial video stream includes at least one video frame; Each video frame in the initial video stream is preprocessed to obtain the video stream to be detected.

14. The method according to claim 13, characterized in that, The preprocessing of each video frame in the initial video stream includes: Resize each video frame in the initial video stream; and / or, Color mode conversion is performed on each video frame in the initial video stream.

15. The method according to any one of claims 1-12, 14, characterized in that, The hand positioning information includes the center coordinates of the hand positioning frame and the width and height of the hand positioning frame.

16. The method according to claim 15, characterized in that, Before performing IOU detection on the hand positioning information in the video stream to be detected, the method further includes: Based on the hand positioning information in each video frame, the size of the hand positioning frame in each video frame is determined; The size of the hand positioning frame in each video frame is compared with a preset size threshold. Based on the comparison results, hands whose hand positioning frame size is smaller than a preset size threshold in the video stream to be detected are filtered out, and the filtered video stream is determined as the video stream to be detected.

17. The method according to any one of claims 1-12, 14, and 16, characterized in that, The method further includes: If the difference in the video frame sequence number in the hand tracking group is not less than a preset difference threshold, or if the IOU of the hand positioning information in the hand tracking group is not greater than a preset overlap threshold, then the hand data detected in the hand tracking group is determined to be different hands.

18. The method according to any one of claims 1-12, 14, and 16, characterized in that, The step of determining the gesture type recognition result of the same hand in the video stream to be detected based on the gesture type in each video frame includes: Obtain the gesture type of the same hand in the current video frame and the initial gesture type recognition results in a preset number of video frames before the current video frame; The initial gesture type recognition result is optimized by using the gesture type in the current video frame to determine the gesture type recognition result of the same hand in the current video frame.

19. The method according to claim 18, characterized in that, The step of optimizing the initial gesture type recognition result using the gesture type in the current video frame to determine the gesture type recognition result of the same hand in the current video frame includes: Based on the gesture type in the current video frame and the initial gesture type recognition results in a preset number of video frames before the current video frame, a gesture type candidate set is formed; The gesture type with the highest frequency is determined from the candidate set of gesture types, and the gesture type with the highest frequency is determined as the gesture type recognition result of the same hand in the current video frame.

20. A gesture recognition device, characterized in that, The gesture recognition device includes a second acquisition unit, a gesture detection unit, an IOU detection unit, and a determination unit, wherein... The second acquisition unit is configured to acquire the video stream to be detected; The gesture detection unit is configured to use a gesture recognition model to perform gesture detection on each video frame in the video stream to be detected, and to determine the gesture type and hand positioning information in each video frame. The IOU detection unit is configured to perform overlap IOU detection on the hand positioning information in the video stream to be detected, and determine the hand detection result of the video stream to be detected. The determining unit is configured to determine the gesture type recognition result of the same hand in the video stream to be detected based on the gesture type in each video frame when the hand detection result indicates that the hand in the video stream to be detected is the same hand. The IOU detection unit is specifically used to determine at least one hand tracking group in the video stream to be detected. Each hand tracking group includes two video frame sequence numbers to be detected for IOU and corresponding hand positioning information. In each hand tracking group, if the difference between the video frame sequence numbers in the hand tracking group is less than a preset difference threshold, and the IOU of the hand positioning information in the hand tracking group is greater than a preset overlap threshold, then the hand data detected in the hand tracking group is determined to be the same hand. If the hand data detected in at least one hand tracking group are all the same hand, then the hand detection result indicates that the hand in the video stream to be detected is the same hand.

21. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein, The memory is used to store computer programs that can run on the processor; The processor is configured to execute the gesture recognition method as described in any one of claims 1-19 when running the computer program.

22. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which, when executed by a processor, implements the gesture recognition method as described in any one of claims 1-19.

Images