Gesture recognition method and device, computer device, and storage medium

Abstract

The application relates to a gesture recognition method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring an image to be recognized; performing feature extraction on the image to be recognized based on a feature extraction network to obtain a target feature map; determining a candidate anchor frame based on a score obtained by predicting a target feature map of a corresponding scale; and recognizing a gesture feature in the candidate anchor frame to obtain a gesture recognition result of the image to be recognized. The method can improve the accuracy of the gesture recognition result.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a gesture recognition method, apparatus, computer device, and storage medium. Background Technology

[0002] With the development of technology, more and more smart terminals are gradually entering users' lives, such as set-top boxes, televisions, computers, mobile phones, and projectors. Gesture recognition technology has been widely applied to various smart terminals, allowing users to control these terminals and perform corresponding functions through gestures. This enables more flexible operation and effectively improves the ease of use. However, traditional gesture recognition methods often suffer from low accuracy. Summary of the Invention

[0003] Therefore, it is necessary to provide a gesture recognition method, device, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of gesture recognition results in response to the above-mentioned technical problems.

[0004] Firstly, this application provides a gesture recognition method. The method includes:

[0005] Acquire the image to be recognized;

[0006] The target feature map is obtained by extracting features from the image to be identified based on a feature extraction network.

[0007] Candidate anchor boxes are determined by predicting the target feature map at the corresponding scale based on the target anchor box score.

[0008] The gesture features in the candidate anchor boxes are identified to obtain the gesture recognition result of the image to be recognized.

[0009] In one embodiment, the step of extracting features from the image to be identified based on a feature extraction network to obtain a target feature map includes:

[0010] The image to be identified is used to extract features based on a feature extraction network to obtain an initial feature map;

[0011] Global average pooling is performed on the initial feature map to obtain the global feature vector corresponding to the initial feature map;

[0012] The global feature vector is processed using a fully connected layer to obtain the weight values ​​corresponding to each channel in the initial feature map;

[0013] The target feature map is obtained based on the weight values ​​and the initial feature map.

[0014] In one embodiment, determining candidate anchor boxes by predicting the target feature map at the corresponding scale based on the target anchor box score includes:

[0015] The score obtained by predicting the target feature map at the corresponding scale using the target anchor box is obtained;

[0016] The target anchor box with the highest score is selected as the candidate anchor box for the current identification.

[0017] In one embodiment, the method further includes:

[0018] The target anchor frame with the highest score is taken as the first target anchor frame, and the intersection-union ratio between the first target anchor frame and each second target anchor frame is calculated sequentially; the second target anchor frames are other target anchor frames besides the first target anchor frame;

[0019] Based on the intersection-union ratio, the next score for the second target anchor frame is determined.

[0020] In one embodiment, determining the next score for the second target anchor frame based on the intersection-union ratio includes:

[0021] If the crossover ratio is less than a preset ratio, the score of the second target anchor frame corresponding to the crossover ratio will be used as the next score of the second target anchor frame;

[0022] If the crossover-union ratio is not less than a preset ratio, the next score of the second target anchor frame is determined by the score of the second target anchor frame corresponding to the crossover-union ratio; the next score of the second target anchor frame is linearly correlated with the score of the second target anchor frame.

[0023] In one embodiment, the method further includes:

[0024] If the highest score of the target anchor frame is higher than the preset score, the target anchor frame with the highest score is taken as the candidate anchor frame for the current identification.

[0025] In one embodiment, the method for determining the target anchor frame includes:

[0026] Get the preset anchor boxes and label boxes;

[0027] The recall rate of the preset anchor box is determined based on the labeled box;

[0028] If the recall rate is higher than the recall rate threshold, the preset anchor frame will be used as a backup anchor frame.

[0029] The spare anchor frames are trained based on the labeled frames;

[0030] If the training convergence condition is met, obtain the offset between the spare anchor box and the annotation box;

[0031] The target anchor frame is determined based on the spare anchor frame and the offset.

[0032] Secondly, this application also provides a gesture recognition device. The device includes:

[0033] The image acquisition module is used to acquire the image to be recognized;

[0034] The feature extraction module is used to extract features from the image to be identified based on a feature extraction network to obtain a target feature map;

[0035] The anchor box determination module is used to determine candidate anchor boxes by predicting the target feature map at the corresponding scale based on the target anchor box.

[0036] The feature recognition module is used to recognize the gesture features in the candidate anchor box and obtain the gesture recognition result of the image to be recognized.

[0037] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0038] Acquire the image to be recognized;

[0039] The target feature map is obtained by extracting features from the image to be identified based on a feature extraction network.

[0040] Candidate anchor boxes are determined by predicting the target feature map at the corresponding scale based on the target anchor box score.

[0041] The gesture features in the candidate anchor boxes are identified to obtain the gesture recognition result of the image to be recognized.

[0042] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0043] Acquire the image to be recognized;

[0044] The target feature map is obtained by extracting features from the image to be identified based on a feature extraction network.

[0045] Candidate anchor boxes are determined by predicting the target feature map at the corresponding scale based on the target anchor box score.

[0046] The gesture features in the candidate anchor boxes are identified to obtain the gesture recognition result of the image to be recognized.

[0047] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0048] Acquire the image to be recognized;

[0049] The target feature map is obtained by extracting features from the image to be identified based on a feature extraction network.

[0050] Candidate anchor boxes are determined by predicting the target feature map at the corresponding scale based on the target anchor box score.

[0051] The gesture features in the candidate anchor boxes are identified to obtain the gesture recognition result of the image to be recognized.

[0052] The aforementioned gesture recognition methods, devices, computer equipment, storage media, and computer program products acquire an image to be recognized, extract features from the image based on a feature extraction network to obtain a target feature map, predict the target feature map at a corresponding scale based on the target anchor boxes, determine candidate anchor boxes based on the predicted scores, and identify gesture features within the candidate anchor boxes to obtain the gesture recognition result for the image to be recognized. This application determines candidate anchor boxes based on the predicted scores from the target feature map at a corresponding scale using the target anchor boxes, thereby identifying gesture features within the candidate anchor boxes. This approach can yield more accurate gesture recognition results for the image to be recognized, improving the accuracy of the gesture recognition results. Attached Figure Description

[0053] Figure 1 This is a diagram illustrating the application environment of a gesture recognition method in one embodiment;

[0054] Figure 2 This is a flowchart illustrating a gesture recognition method in one embodiment;

[0055] Figure 3 This is a flowchart illustrating step 204 in one embodiment;

[0056] Figure 4 This is a schematic diagram illustrating the process of determining the target anchor frame in one embodiment;

[0057] Figure 5 This is a model structure diagram of a YOLOv5 network model in one embodiment;

[0058] Figure 6 This is a schematic diagram of the feature extraction network structure in one embodiment;

[0059] Figure 7 This is a schematic diagram of the feature extraction network structure in another embodiment;

[0060] Figure 8 This is a structural block diagram of a gesture recognition device in one embodiment;

[0061] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] The gesture recognition method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Server 104 receives the image to be recognized from terminal 102, extracts features from the image based on a feature extraction network to obtain a target feature map, predicts the target feature map at the corresponding scale based on the target anchor boxes, determines candidate anchor boxes, identifies gesture features within the candidate anchor boxes, and obtains the gesture recognition result for the image to be recognized. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster composed of multiple servers.

[0064] The gesture recognition method provided in this application is not limited to the application scenarios of terminal-server interaction described above, but can also be applied to either the terminal or the server independently.

[0065] In one embodiment, such as Figure 2 As shown, a gesture recognition method is provided, which can be applied to... Figure 1 The following steps, from step 202 to step 208, will be used as an example to illustrate the process.

[0066] Step 202: Obtain the image to be recognized.

[0067] The server can acquire images to be recognized sent by electronic devices, or images to be recognized captured by the server through a camera. These images include gesture features, which can be one or more. For example, an image to be recognized may contain gesture features corresponding to two different gestures. In a practical application scenario, an electronic device captures an image to be recognized using its camera and sends it to the server, which then acquires the image.

[0068] Step 204: Extract features from the image to be recognized using a feature extraction network to obtain the target feature map.

[0069] The server extracts features from the image to be recognized using a feature extraction network to obtain a target feature map. This feature extraction network can be a feature extraction algorithm or a machine learning-based feature extraction model, such as the Backbone and Neck layers of a YOLOv5 network model. The feature extraction network can also include a convolutional (conv) network. In this embodiment, the server extracts gesture features from the image to be recognized using the feature extraction network to obtain the target feature map.

[0070] Optionally, the server performs feature extraction on the image to be recognized based on a feature extraction network to obtain corresponding multi-scale feature maps. For example, by using a YOLOv5 network model to extract features from the image to be recognized, three or four target feature maps of different scales can be obtained. Assuming the image to be recognized is 640*640 pixels, feature extraction is performed on the image to be recognized to obtain feature maps of three scales: 20*20 pixels, 40*40 pixels, and 80*80 pixels. Furthermore, the 80*80 pixel feature map can be upsampled to obtain a 160*160 pixel feature map, thus obtaining feature maps of four scales: 20*20 pixels, 40*40 pixels, 80*80 pixels, and 160*160 pixels.

[0071] Step 206: Based on the score obtained by predicting the target feature map at the corresponding scale using the target anchor box, determine the candidate anchor box.

[0072] The server determines candidate anchor boxes by predicting scores from target feature maps of corresponding scales based on a preset number of target anchor boxes. These target anchor boxes can be initially set preset anchor boxes, or preset anchor boxes selected based on recall calculations performed on labeled anchor boxes, where the recall rate exceeds a threshold. The number of target anchor boxes can be set as needed. Target anchor boxes of different scales correspond to target feature maps of different scales, and target anchor boxes of the same scale correspond to target feature maps of the same scale. Conversely, smaller-scale target anchor boxes correspond to larger-scale target feature maps for detecting smaller targets, and larger-scale target anchor boxes correspond to smaller-scale target feature maps for detecting larger targets. For example, there are 12 target anchor frames, including four sizes, with three sizes in each size category. For example, the first size category is [5,6], [8,14], [15,11], the second size category is [10,13], [16,30], [33,23], the third size category is [30,61], [62,45], [59,119], and the fourth size category is [116,90], [156,198], [373,326]. The values ​​x and y in [x,y] are used to represent the length and width of the target anchor frame, respectively. For example, [5,6] represents that the length of the target anchor frame is 5 pixels and the width is 6 pixels. The target feature maps have four scales: 20*20 pixels, 40*40 pixels, 80*80 pixels, and 160*160 pixels. The target anchor boxes of the first size are distributed on the target feature map of the 160*160 pixel scale for feature prediction, the target anchor boxes of the second size are distributed on the target feature map of the 80*80 pixel scale for feature prediction, the target anchor boxes of the third size are distributed on the target feature map of the 40*40 pixel scale for feature prediction, and the target anchor boxes of the fourth size are distributed on the target feature map of the 20*20 pixel scale for feature prediction.

[0073] Understandably, the score obtained by predicting gesture features in the target feature map based on the target anchor boxes can be characterized by the confidence level of the prediction of gesture features in the target feature map using the target anchor boxes. In this embodiment, candidate anchor boxes can be determined by predicting scores based on the features of all target anchor boxes.

[0074] Optionally, the target anchor box with the highest score can be used as the candidate anchor box for the current identification.

[0075] Step 208: Identify gesture features in the candidate anchor boxes to obtain the gesture recognition result of the image to be recognized.

[0076] The server identifies gesture features within candidate anchor boxes to obtain gesture recognition results for the image to be recognized. Understandably, by identifying gesture features within candidate anchor boxes, the recognition range is reduced, thereby improving the probability and efficiency of gesture feature recognition. Optionally, by identifying gesture features within candidate anchor boxes to obtain gesture information, the category of the gesture information is identified, and regression is performed on the corresponding category of gesture information to obtain the position and size of the gesture information, thus obtaining the corresponding gesture recognition result. The gesture recognition result may include gesture type and gesture name, and may also include information such as gesture position and gesture size.

[0077] In the above gesture recognition method, an image to be recognized is acquired, and features are extracted from the image using a feature extraction network to obtain a target feature map. Based on the target anchor boxes, a score is predicted on the target feature map at the corresponding scale to determine candidate anchor boxes. Gesture features within the candidate anchor boxes are then recognized to obtain the gesture recognition result for the image to be recognized. Determining candidate anchor boxes based on the predicted scores of the target feature map at the corresponding scale, and thus recognizing the gesture features within the candidate anchor boxes, can yield more accurate gesture recognition results for the image to be recognized, improving the accuracy of gesture recognition.

[0078] In one embodiment, such as Figure 3 As shown, step 204, which extracts features from the image to be identified based on a feature extraction network to obtain a target feature map, includes steps 302 to 308.

[0079] Step 302: Extract features from the image to be recognized using a feature extraction network to obtain an initial feature map.

[0080] The feature extraction network extracts features from the image to be recognized, resulting in multi-scale initial feature maps. This can be understood as the initial feature maps being directly extracted by the feature extraction network without any further processing.

[0081] Step 304: Perform global average pooling on the initial feature map to obtain the global feature vector corresponding to the initial feature map.

[0082] In this embodiment, global average pooling is performed on the initial feature map to obtain the global feature vector corresponding to the initial feature map. For example, through global average pooling, the two-dimensional features (width and height) of each channel are compressed into a real number, and the initial feature map [H,W,C] is transformed into [H,W,1], thus obtaining the global feature vector corresponding to the initial feature map.

[0083] Step 306: Process the global feature vector based on the fully connected layer to obtain the weight values ​​corresponding to each channel in the initial feature map.

[0084] The global feature vector is processed using fully connected layers to obtain the weight values ​​corresponding to each channel in the initial feature map. Optionally, the global feature vector can be processed using multiple fully connected layers, such as two fully connected layers. By using two fully connected layers, the correlation between each channel of the global feature vector can be constructed, and the weight values ​​corresponding to each channel in the initial feature map can be determined. The obtained weight values ​​can be normalized to obtain weight values ​​between 0 and 1.

[0085] Step 308: Obtain the target feature map based on the weight values ​​and the initial feature map.

[0086] The server can obtain the target feature map from the weight values ​​and the initial feature map. Optionally, the target feature map can be obtained by multiplying the weight values ​​and the initial feature map. Specifically, the target feature map can be obtained by multiplying the weight values ​​corresponding to each channel by the corresponding channel features in the initial feature map.

[0087] In the above embodiments, a feature extraction network is used to extract features from the image to be recognized, obtaining an initial feature map. Global average pooling is then performed on the initial feature map to obtain a global feature vector corresponding to the initial feature map. A fully connected layer processes the global feature vector to obtain the weight values ​​corresponding to each channel in the initial feature map. Based on the weight values ​​and the initial feature map, a target feature map is obtained. By increasing the weight values ​​of each channel to their corresponding channels, the features of channels with larger weight values ​​become more prominent, while the features of channels with smaller weight values ​​are weakened. This highlights the features of important channels, making the gesture features in the target feature map more obvious and easier to recognize, thus improving the accuracy of gesture recognition results.

[0088] In some embodiments, determining candidate anchor boxes based on the score obtained by predicting the target feature map at the corresponding scale using the target anchor box includes: obtaining the score obtained by predicting the target feature map at the corresponding scale using the target anchor box, and taking the target anchor box with the highest score as the candidate anchor box for the current identification.

[0089] In this embodiment, gesture features are predicted from the target feature map using target anchor boxes. Each target anchor box receives a corresponding prediction score, which represents the accuracy of the prediction. The higher the prediction score, the higher the accuracy of the prediction; the lower the prediction score, the lower the accuracy of the prediction. Selecting the target anchor box with the highest score as the candidate anchor box for the current recognition can improve the accuracy of the gesture recognition results.

[0090] Understandably, gesture features in the target feature map can be identified multiple times based on candidate anchor boxes to obtain multiple gesture recognition results, thereby improving the accuracy of gesture recognition. Optionally, the gesture results from multiple recognitions can be gestures of the same category or gestures of different categories, with one category of gesture being recognized each time.

[0091] In some embodiments, the gesture recognition method further includes: taking the target anchor box with the highest score as the first target anchor box, sequentially calculating the intersection-union ratio (IUU) between the first target anchor box and each second target anchor box, wherein the second target anchor box is any target anchor box other than the first target anchor box; and determining the next score of the second target anchor box based on the IUU.

[0092] In this embodiment, the target anchor box with the highest score is used as the candidate anchor box for the current identification. Simultaneously, the target anchor box with the highest score can be used as the first target anchor box. The intersection-union ratio (IUR) between the first target anchor box and each second target anchor box is calculated sequentially. Here, the second target anchor boxes are all target anchor boxes other than the first target anchor box. The IUR is the result of dividing the overlapping area of ​​two regions by the union of the two regions; it is a standard for measuring the accuracy of detecting corresponding objects on a specific dataset. In this embodiment, the IUR can be the ratio between the overlapping area of ​​the first and second target anchor boxes and the union of the first and second target anchor boxes. Furthermore, based on the IUR, the next score for the corresponding second target anchor box can be determined. For each IUR, the next score for the second target anchor box corresponding to that IUR can be determined. Understandably, the first target anchor box with the highest score is used as a candidate anchor box for the current identification and does not participate in the next identification.

[0093] Optionally, the next score for the second target anchor frame can be determined based on the relationship between the intersection-union ratio (IU) and a preset ratio. For example, if the IU is less than the preset ratio, the current score of the corresponding second target anchor frame is used as the next score for that second target anchor frame; if the IU is not less than the preset ratio, the next score for the corresponding second target anchor frame is set to zero.

[0094] In the above embodiments, the next score of other target anchor boxes is determined based on the intersection-union ratio between the highest-scoring target anchor box and other target anchor boxes. That is, the next score of other target anchor boxes is determined based on the positional relationship between the highest-scoring target anchor box and other target anchor boxes in the current identification, which can obtain a more accurate score for other target anchor boxes in the next identification.

[0095] In some embodiments, determining the next score for the second target anchor frame based on the cross-union ratio includes:

[0096] If the cross-union ratio is less than a preset ratio, the current score of the second target anchor frame corresponding to the cross-union ratio is used as the next score of the second target anchor frame; if the cross-union ratio is not less than the preset ratio, the next score of the second target anchor frame is determined by the current score of the second target anchor frame corresponding to the cross-union ratio; wherein, the next score of the second target anchor frame is linearly correlated with the current score of the second target anchor frame.

[0097] In this embodiment, the next score of the second target anchor frame can be determined based on the cross-union ratio (CUI) and the current score of the second target anchor frame. If the CUI is less than a preset ratio, the current score of the second target anchor frame corresponding to the CUI is used as the next score of the second target anchor frame; if the CUI is not less than the preset ratio, the next score of the second target anchor frame is determined by the current score of the second target anchor frame corresponding to the CUI. The next score of the second target anchor frame is linearly correlated with the current score of the second target anchor frame; for example, the next score of the second target anchor frame is a function value with the current score of the second target anchor frame as the independent variable.

[0098] Optionally, if the crossover-union ratio is not less than a preset ratio, the product of the current score of the second target anchor frame and the corresponding crossover-union ratio can be calculated, and the difference between the current score of the second target anchor frame and the product can be used as the next score of the second target anchor frame.

[0099] In one example, the calculation method for the next score of the second target anchor box is detailed in the following formula (1).

[0100]

[0101] In formula (1), s i+1 Indicates the next score for the second target anchor box, s i This represents the current score of the second target anchor box, iou(M,b) i ) indicates the second target anchor frame b i The intersection-over-union ratio (CLOUD) between N and the first target anchor frame M t This indicates the preset ratio.

[0102] In the above embodiments, the next score of the second target anchor box corresponding to the intersection-union ratio (IU) is determined based on the relationship between the IU and a preset ratio. If the IU is less than the preset ratio, the current score of the second target anchor box corresponding to the IU is used as the next score of the second target anchor box. If the IU is not less than the preset ratio, the next score of the second target anchor box is determined by the current score of the second target anchor box corresponding to the IU. The next score of the second target anchor box is linearly correlated with the current score. This can sequentially filter out second target anchor boxes with large intersections with the first target anchor box with the highest score, preventing candidate anchor boxes determined in different iterations from being the same target anchor box. The next score of each second target anchor box is determined sequentially, thereby determining more accurate candidate anchor boxes.

[0103] In one embodiment, the method further includes: if the highest score of the target anchor box is higher than the preset score, the target anchor box with the highest score is taken as the candidate anchor box for the current identification.

[0104] In this embodiment, if the highest score of the target anchor box is higher than the preset score, the target anchor box with the highest score is selected as the candidate anchor box for the current recognition. Optionally, if the highest score of the target anchor box is not higher than the preset score, the gesture feature recognition process ends.

[0105] In one example, the score obtained by predicting the target feature map at the corresponding scale using the target anchor box is obtained. If the score corresponding to anchor box A is the highest among all target anchor boxes, then the score of anchor box A is compared with the preset score. If the score of anchor box A is higher than the preset score, then anchor box A with the highest score is taken as the candidate anchor box for the current identification. Similarly, when filtering the next candidate anchor box, the score corresponding to the current target anchor box is obtained. If the score of anchor box B is the highest among all target anchor boxes, then anchor box B is taken as the candidate anchor box for the current identification if the score of anchor box B is higher than the preset score.

[0106] In the above embodiments, the target anchor box with the highest score is used as the candidate anchor box for the current recognition only when the highest score of the target anchor box is higher than the preset score, so as to ensure that the features in the candidate anchor box are closer to the gesture features, thereby further improving the accuracy of the gesture recognition results.

[0107] In one embodiment, such as Figure 4 As shown, the method for determining the target anchor frame includes the following steps 402 to 412.

[0108] Step 402: Obtain the preset anchor box and annotation box.

[0109] The process involves acquiring a preset number of preset anchor boxes and labeled boxes from all labeled images. The size of the preset anchor boxes can be set empirically, their positions can be randomly distributed, and the preset number can be set according to the actual application scenario; for example, the preset number could be 9. The preset anchor boxes include 3 types, i.e., 3 sizes, corresponding to the detection of gesture features in gesture feature maps at three scales. Labeled boxes are the boxes in the labeled images that annotate gesture features. These labels can be obtained manually or automatically using annotation tools. Acquiring the labeled boxes includes obtaining their positions and sizes within the target gesture image.

[0110] Step 404: Determine the recall rate of the preset anchor box based on the annotation box.

[0111] In this embodiment, the recall rate of the preset anchor frame is determined by the annotation box. The recall rate of the preset anchor frame can be determined based on the ratio of the length and width of the annotation box to the length and width of the preset anchor frame, respectively.

[0112] In one example, suppose there are n bounding boxes and 9 preset anchor boxes. The recall rate of the 9 preset anchor boxes is determined using the n bounding boxes. The ratios of the length and width of the n bounding boxes to the length and width of each preset anchor box are calculated, resulting in n*9 sets of length and width ratios. The minimum ratio is determined from each set, and the maximum ratio is determined from the minimum ratio corresponding to each preset anchor box, resulting in n maximum ratios. The number of ratios greater than a ratio threshold is determined from the n maximum ratios. The ratio of the number of ratios greater than the ratio threshold to n is used as the recall rate of the 9 preset anchor boxes. The ratio threshold can be set according to the actual application scenario. The ratio threshold is used to characterize the minimum matching degree between the preset anchor boxes and the bounding boxes. The calculation process in this example can be seen in the following formulas (2) to (3).

[0113]

[0114]

[0115] Step 406: If the recall rate is higher than the recall rate threshold, the preset anchor box is used as the backup anchor box.

[0116] If the recall rate is higher than the recall rate threshold, preset anchor boxes are used as backup anchor boxes, which are boxes used by the object recognition model to recognize gesture features. Optionally, if the recall rate is not higher than the preset recall rate, backup anchor boxes can be calculated using a genetic algorithm and a K-means algorithm.

[0117] Step 408: Train the spare anchor boxes based on the labeled boxes.

[0118] Step 410: If the training convergence condition is met, obtain the offset between the spare anchor box and the annotation box.

[0119] Step 412: Determine the target anchor frame based on the spare anchor frame and the offset.

[0120] The backup anchor boxes are trained using labeled boxes, allowing them to learn the features of the labeled boxes. When the training convergence condition is met, the offset between the backup anchor boxes and the labeled boxes is obtained. This offset characterizes the degree to which the backup anchor box deviates from the labeled box. Based on the backup anchor boxes and the offset, the target anchor box can be determined. Optionally, the product of the backup anchor box and the offset can be used as the target anchor box.

[0121] In the above embodiments, the recall rate of the preset anchor box is determined based on the labeled bounding box. Based on the relationship between the recall rate of the preset anchor box and the recall rate threshold, it is determined whether the preset anchor box should be used as a backup anchor box. If the recall rate is higher than the preset recall rate, the preset anchor box is used as a backup anchor box, and the backup anchor box is trained based on the labeled bounding box. If the training convergence condition is met, the offset between the backup anchor box and the labeled bounding box is obtained. The target anchor box is determined based on the backup anchor box and the offset. This can determine a more accurate target anchor box, improve the accuracy of candidate anchor boxes, and thus improve the accuracy of gesture recognition.

[0122] In one embodiment, a gesture recognition model is provided to implement the gesture recognition method described above. The gesture recognition model may be, for example, a YOLOv5 network model, or a model trained and improved based on the YOLOv5 network model. The model structure of the YOLOv5 network model is as follows: Figure 5As shown. The YOLOv5 network model is mainly divided into four modules: (1) Input end: The input end is used to receive the target gesture image. The size of the target gesture image is uniformly 608*608 pixels. (2) Backbone: YOLOv5 uses CSP Darknet53 structure and Focus structure as Backbone. CSP Darknet53 is composed of CSP and Darknet53. The problem of excessive inference computation is caused by the repetition of gradient information in network optimization. Therefore, the CSP module is used to divide the feature mapping of the base layer into two parts, and then merge them through cross-stage hierarchical structure. While reducing the amount of computation, the accuracy can be guaranteed. Darknet53 is a convolutional neural network, mainly used to extract rich feature information in the input image. The role of Focus is to concentrate the W and H information on the channel in the downsampling of the image without losing information, and then use 3×3 convolution for feature extraction, so that the feature extraction is more sufficient. Although it increases the amount of computation, it retains more complete image downsampling information for subsequent feature extraction. (3) Neck: SPP+FPN+PAN. SPP: Replaces the conventional pooling layer after the convolutional layer, which can increase the receptive field and better acquire multi-scale features. FPN is downsampling and conveys features from top to bottom, while PAN is upsampling and conveys features from bottom to top. The PAN module performs crazy fusion of features at different levels. It adds a bottom-up feature pyramid structure on the basis of the FPN module. The feature maps of the top-down part and the bottom-up part are fused to obtain the final feature map, which further improves the overall feature extraction capability. (4) Output end: Used to perform target detection on the feature pyramid. The output end includes some convolutional layers, pooling layers and fully connected layers. In the YOLOv5 network model, the detection head module is mainly responsible for multi-scale target detection on the feature maps extracted by the backbone network. This module mainly includes Anchors, which are used to define target boxes of different sizes and aspect ratios. Usually, K-means clustering is used to cluster the target boxes of the training set. It can be calculated before model training and stored in the model for generating detection boxes during prediction. Classification is used to classify each detection box to determine whether it is a target object. It uses fully connected layers and a softmax function to classify the features. Regression is used to regress each detection box to obtain its position and size, typically using fully connected layers to regress the features. Backbone and Neck are feature extraction networks in the YOLOv5 network model, used to extract gesture features from the image to be recognized, obtaining the target feature map.

[0123] Optionally, a small object detection layer can be added to the YOLOv5 network model. This layer predicts gesture features in a larger-scale target feature map based on smaller-scale target anchor boxes, thereby detecting small-scale gesture features. This allows the YOLOv5 network model to focus more on detecting small-scale gesture features, improving the accuracy of gesture feature prediction. For example, the YOLOv5 network model has 9 target anchor boxes, including three sizes, with 3 boxes in each size. The input image to be recognized is 640*640 pixels. The first size is [10,13], [16,30], [33,23]; the second size is [30,61], [62,45], [59,119]; and the third size is [116,90], [156,198], [373,326]. Then, the target anchor boxes of the first size are distributed in 8... Feature prediction is performed on a target feature map with a scale of 0*80 pixels to detect gesture features corresponding to the first size. For the second size, target anchor boxes are distributed on a target feature map with a scale of 40*40 pixels for feature prediction to detect gesture features corresponding to the second size. For the third size, target anchor boxes are distributed on a target feature map with a scale of 20*20 pixels for feature prediction to detect gesture features corresponding to the third size. The feature extraction network structure in the YOLOv5 network model is as follows: Figure 6 As shown. To add another layer to detect gesture features at a smaller scale, a target feature map can be added on top of the 80*80 pixel target feature map. Since the YOLOv5 network model Neck does not have a 160*160 pixel target feature map, the 80*80 pixel target feature map is upsampled to obtain a 160*160 pixel target feature map. This 160*160 pixel target feature map can then be concatenated with the 160*160 pixel target feature map in the backbone to obtain the final 160*160 pixel target feature map used for detection. The upsampled feature extraction network structure is shown below. Figure 7As shown. Correspondingly, target anchor boxes of corresponding sizes need to be added, for example, [5,6], [8,14], [15,11], distributed on a target feature map of 160*160 pixels for feature prediction, used to detect gesture features of the corresponding size. That is, the first type of size is [5,6], [8,14], [15,11], the second type of size is [10,13], [16,30], [33,23], the third type of size is [30,61], [62,45], [59,119], and the fourth type of size is [116,90], [156,198], [373,326]. Then, the target anchor boxes of the first type of size are distributed on a target feature map of 160*160 pixels for feature prediction, used to detect the first size feature corresponding to the first type of size; the target anchor boxes of the second type of size are distributed on a target feature map of 8 ... Feature prediction is performed on the target feature map at a scale of 0*80 pixels to detect the second-size features corresponding to the second-size category; the target anchor boxes of the third-size category are distributed on the target feature map at a scale of 40*40 pixels to perform feature prediction to detect the third-size features corresponding to the third-size category; the target anchor boxes of the fourth-size category are distributed on the target feature map at a scale of 20*20 pixels to perform feature prediction to detect the fourth-size features corresponding to the fourth-size category. Among these, the first-size features are smaller than the second-size features, the second-size features are smaller than the third-size features, and the third-size features are smaller than the fourth-size features.

[0124] Optionally, the weight values ​​for each channel in the initial feature map can be determined using the Squeeze Excitation (SE) attention mechanism. The initial feature map is obtained by extracting features from the image to be recognized using the feature extraction network in the YOLOv5 network model. Adding weight values ​​to the corresponding channels makes it easier to focus on features from channels with higher information content, while suppressing features from channels with lower information content.

[0125] Optionally, a modified Non-Maximum Suppression (NMS) algorithm can be used to filter the target anchor boxes to obtain candidate anchor boxes. The target anchor box with the highest score is taken as the first target anchor box, and the intersection-union ratio (IUR) between the first target anchor box and each second target anchor box is calculated sequentially, where the second target anchor boxes are all target anchor boxes other than the first target anchor box; the next score of the second target anchor box is determined based on the IUR. For example, the next score of the second target anchor box can be calculated using formula (1).

[0126] Optionally, the C3 structure in the YOLOv5 backbone can be replaced with the heavily weighted MobileNetV3 network. The C3 structure in the backbone is a CSP structure. The YOLOv5 backbone feature extraction network uses a C3 structure, resulting in a large number of parameters, slower detection speed, and limited applications. In some practical applications, such as mobile or embedded devices, large and complex models are difficult to apply, potentially leading to insufficient memory, slow response time, and high latency. Therefore, in these practical applications, the backbone feature extraction network can be replaced with the lighter MobileNetV3 network to achieve model lightweighting and balance speed and accuracy.

[0127] Optionally, the γ in the gesture recognition model can be normalized to obtain a normalized γ. The normalized γ can then be regularized to obtain a corresponding sparse matrix. Values ​​in the sparse matrix that are less than a parameter threshold can be set to zero to obtain an optimized sparse matrix. This leads to an optimized YOLOv5 network model. By retaining channels with larger γ and deleting channels with smaller γ (channels with smaller γ correspond to less important information), pruning of the gesture recognition model can be achieved, reducing the model's complexity while ensuring its accuracy.

[0128] Optionally, the precision of the gesture recognition model's parameters can be converted from a first precision to a second precision, where the second precision is lower than the first precision. The gesture recognition model corresponding to the second-precision model parameters is then used as the optimized gesture recognition model. For example, the precision of the model parameters can be converted from a 32-bit float type to an 8-bit integer int8. Converting the model parameters from high precision to low precision can compress the model parameters, reduce memory usage, lower device power consumption, and speed up operation.

[0129] In some practical application scenarios, the first step is to convert the gesture recognition model trained in PyTorch into an ONNX model. Then, the ONNX model is parsed to generate an inference engine. PyTorch is a deep learning framework, and ONNX is an open file format designed for machine learning to store trained models, allowing different AI frameworks to store model data and interact using the same format. Next, the ONNX file is converted into an NCNN file, yielding the model document file and its binary file required by NCNN. After model loading, it can be ported to operating systems such as Android and iOS. Then, the NCNN file is used to generate an APK (Android Package) for deployment to electronic devices, such as smartphones. Taking Android as an example, the compiler compiles various source code files in the Android project, obtaining compiled resources and dex files. These compiled resources and dex files are then passed to a packager, which performs APK signing. The zipalign optimization tool in the Android SDK is used to optimize the APK signing, improving the interaction efficiency between the optimized application and the Android operating system. The APK file is then installed on the Android operating system, generating a gesture recognition app on the corresponding electronic device. The DEX file is the executable file of the Android system, containing all the application's operation instructions and runtime data. After a Java program is compiled into class files, the dx tool is used to integrate all the class files into a single DEX file. This allows the classes to share data, reduces redundancy to some extent, and makes the file structure more compact. The APK, or Android installation package, is a format adapted for Android applications. An APK file is essentially a ZIP archive, packaging the Android SDK-compiled project into an installer file supported by the Android system.

[0130] In one embodiment, a server or terminal device acquires an image to be recognized, extracts features from the image using a feature extraction network to obtain a target feature map, predicts a score based on the target anchor box at the corresponding scale of the target feature map, determines candidate anchor boxes, identifies gesture features within the candidate anchor boxes, and obtains the gesture recognition result for the image to be recognized. When communicating with a projection device, the device is instructed to execute the target operation command corresponding to the gesture recognition result. The projection device stores a mapping relationship between the gesture recognition result and the target operation command. This allows for efficient control of the projection device based on different gesture recognition results, avoiding untimely control due to the inconvenience of obtaining remote controls or other auxiliary accessories, and improving the control efficiency of the projection device.

[0131] In a practical application scenario, electronic devices and projection devices communicate via Bluetooth. An app on the electronic device is opened, instructing the camera to capture an image of the target gesture or retrieve it from the photo album. The app then uses a gesture recognition model to detect gesture features in the target feature map based on candidate anchor boxes, obtaining the gesture recognition result. This result is sent to the projection device, instructing it to execute the corresponding target operation command. The projection device maintains a one-to-one mapping between gesture recognition results and target operation commands. For example, a thumbs-down gesture corresponds to a volume down gesture; a thumbs-up gesture to a volume up gesture; an index finger pointing left to a back gesture; an index finger pointing right to a forward gesture; five fingers spread to a pause gesture; and an "OK" gesture to continue, and so on.

[0132] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0133] Based on the same inventive concept, this application also provides a gesture recognition device for implementing the gesture recognition method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more gesture recognition device embodiments provided below can be found in the limitations of the gesture recognition method described above, and will not be repeated here.

[0134] In one embodiment, such as Figure 8 As shown, a gesture recognition device is provided, including: an image acquisition module 802, a feature extraction module 804, an anchor box determination module 806, and a feature recognition module 808, wherein:

[0135] Image acquisition module 802 is used to acquire the image to be recognized;

[0136] Feature extraction module 804 is used to extract features from the image to be identified based on a feature extraction network to obtain a target feature map;

[0137] Anchor frame determination module 806 is used to determine candidate anchor frames based on the score obtained by predicting the target feature map at the corresponding scale based on the target anchor frame.

[0138] The feature recognition module 808 is used to recognize the gesture features in the candidate anchor box and obtain the gesture recognition result of the image to be recognized.

[0139] In one embodiment, the feature extraction module 804 is further configured to:

[0140] The image to be identified is subjected to feature extraction based on a feature extraction network to obtain an initial feature map; global average pooling is performed on the initial feature map to obtain a global feature vector corresponding to the initial feature map; the global feature vector is processed based on a fully connected layer to obtain the weight value corresponding to each channel in the initial feature map; the target feature map is obtained based on the weight value and the initial feature map.

[0141] In one embodiment, the anchor frame determining module 806 is further configured to:

[0142] The target anchor box is used to predict the target feature map at the corresponding scale and the score is obtained; the target anchor box with the highest score is used as the candidate anchor box for the current identification.

[0143] In one embodiment, the gesture recognition device further includes a scoring determination module, used for:

[0144] The target anchor frame with the highest score is taken as the first target anchor frame, and the intersection-union ratio (CUI) between the first target anchor frame and each second target anchor frame is calculated sequentially; the second target anchor frames are all other target anchor frames except the first target anchor frame; the next score of the second target anchor frame is determined based on the CUI.

[0145] In one embodiment, the determining module is further configured to:

[0146] If the cross-union ratio is less than a preset ratio, the score of the second target anchor frame corresponding to the cross-union ratio is used as the next score of the second target anchor frame; if the cross-union ratio is not less than the preset ratio, the next score of the second target anchor frame is determined by the score of the second target anchor frame corresponding to the cross-union ratio; the next score of the second target anchor frame is linearly correlated with the score of the second target anchor frame.

[0147] In one embodiment, the gesture recognition device further includes a scoring management module, used to: when the highest score of the target anchor frame is higher than a preset score, use the target anchor frame with the highest score as a candidate anchor frame for the current recognition.

[0148] In one embodiment, the target anchor frame module is used to implement the method of determining the target anchor frame, including:

[0149] Obtain a preset anchor box and a labeled box; determine the recall rate of the preset anchor box based on the labeled box; if the recall rate is higher than the recall rate threshold, use the preset anchor box as a backup anchor box; train the backup anchor box based on the labeled box; if the training convergence condition is met, obtain the offset between the backup anchor box and the labeled box; determine the target anchor box based on the backup anchor box and the offset.

[0150] Each module in the aforementioned gesture recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0151] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media to run. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a gesture recognition method.

[0152] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0153] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the gesture recognition method described above.

[0154] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the gesture recognition method described above.

[0155] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the gesture recognition method described above.

[0156] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0157] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0158] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0159] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A gesture recognition method, characterized in that, The method includes: Acquire the image to be recognized; Based on a feature extraction network, feature extraction is performed on the image to be identified to obtain a multi-scale target feature map; The target anchor box with the highest score among the predicted scores obtained by predicting the target feature map of the corresponding scale is used as the candidate anchor box for the current identification; the small-scale target anchor box corresponds to the large-scale target feature map, and the large-scale target anchor box corresponds to the small-scale target feature map; the target anchor box includes the initially set preset anchor box. The gesture features in the candidate anchor frame are identified to obtain the gesture recognition result of the image to be recognized, and the projection device is instructed to execute the target operation command corresponding to the gesture recognition result.

2. The method according to claim 1, characterized in that, The feature extraction network is used to extract features from the image to be identified, resulting in a multi-scale target feature map, including: The image to be identified is used to extract features based on a feature extraction network to obtain an initial feature map; Global average pooling is performed on the initial feature map to obtain the global feature vector corresponding to the initial feature map; The global feature vector is processed using a fully connected layer to obtain the weight values ​​corresponding to each channel in the initial feature map; Based on the weight values ​​and the initial feature map, a multi-scale target feature map is obtained.

3. The method according to claim 1, characterized in that, The method further includes: The target anchor frame with the highest score is taken as the first target anchor frame, and the intersection-union ratio between the first target anchor frame and each second target anchor frame is calculated sequentially; the second target anchor frames are other target anchor frames besides the first target anchor frame; Based on the intersection-union ratio, the next score for the second target anchor frame is determined.

4. The method according to claim 3, characterized in that, Determining the next score for the second target anchor frame based on the intersection-union ratio includes: If the crossover ratio is less than a preset ratio, the current score of the second target anchor frame corresponding to the crossover ratio will be used as the next score of the second target anchor frame. If the crossover-union ratio is not less than a preset ratio, the next score of the second target anchor frame is determined by the current score of the second target anchor frame corresponding to the crossover-union ratio; the next score of the second target anchor frame is linearly correlated with the current score of the second target anchor frame.

5. The method according to claim 4, characterized in that, If the intersection-union ratio is not less than a preset ratio, the next score of the second target anchor frame is determined based on the current score of the second target anchor frame corresponding to the intersection-union ratio, including: If the crossover-union ratio is not less than a preset ratio, calculate the product of the current score of the second target anchor frame corresponding to the crossover-union ratio and the crossover-union ratio, and take the difference between the current score of the second target anchor frame and the product as the next score of the second target anchor frame.

6. The method according to claim 1, characterized in that, The method further includes: If the highest score of the target anchor frame is higher than the preset score, the target anchor frame with the highest score is taken as the candidate anchor frame for the current identification.

7. The method according to claim 1, characterized in that, The method for determining the target anchor frame includes: Get the preset anchor boxes and label boxes; The recall rate of the preset anchor box is determined based on the labeled box; If the recall rate is higher than the recall rate threshold, the preset anchor frame will be used as a backup anchor frame. The spare anchor boxes are trained based on the labeled boxes; If the training convergence condition is met, obtain the offset between the spare anchor box and the annotation box; The target anchor frame is determined based on the spare anchor frame and the offset.

8. A gesture recognition device, characterized in that, The device includes: The image acquisition module is used to acquire the image to be recognized; The feature extraction module is used to extract features from the image to be identified based on a feature extraction network to obtain a multi-scale target feature map; An anchor frame determination module is used to select the target anchor frame with the highest score among the scores obtained by predicting the target anchor frame to the target feature map of the corresponding scale as the candidate anchor frame for the current identification; the small-scale target anchor frame corresponds to the large-scale target feature map, and the large-scale target anchor frame corresponds to the small-scale target feature map; the target anchor frame includes the initially set preset anchor frame. The feature recognition module is used to recognize the gesture features in the candidate anchor frame, obtain the gesture recognition result of the image to be recognized, and instruct the projection device to execute the target operation command corresponding to the gesture recognition result.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images