Data labeling method and device for hand detection, equipment and storage medium
By using multiple hand detection models to detect images and combining their detection results, the problem of hand detection frame jitter is solved, achieving more stable hand detection and more accurate gesture recognition.
Patent Information
- Application Number
- CN202311434443.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-31
- Publication Date
- 2025-05-02
AI Technical Summary
The existing hand detection algorithm is prone to jitter in the hand detection frame when recognizing gestures, resulting in jitter in the hand key point detection algorithm on the timing screen, which brings great errors to the recognition results.
By using multiple hand detection models to detect images separately, the intersecting ratio of the hand detection box in the detection results of each model is calculated. If the intersecting ratio reaches a preset threshold, the detection results of each model are combined to generate labeling data, reducing labeling errors and improving the stability of the detection box.
By combining the detection results of multiple hand detection models, it is possible to reduce labeling errors, reduce jitter of the hand detection frame, and improve the stability of the hand detection frame, thereby improving the accuracy and stability of gesture recognition.
Smart Images
Figure CN119919990A_ABST
Abstract
Description
Technical Field
[0001] The disclosed embodiments relate to the field of gesture control technology, and in particular to a data annotation method, device, equipment and storage medium for hand detection. Background Art
[0002] With the development of vehicle technology, the demand for controlling the rear entertainment screen of the vehicle through gestures has increased. At present, in order to recognize gestures, it is necessary to first collect images containing gestures and input the images into the hand detection algorithm. The hand detection algorithm can detect the position of the hand in the image. For example, the hand is selected with a detection box in the image, and the position of the detection box in the image indicates the position of the hand in the image. Then, the position of the key points of the hand is detected by the hand key point detection algorithm, and then the dynamic gestures are judged, and then the actions such as waving up and down, clenching fists, and opening are recognized to control the return, click, release and other functions on the screen.
[0003] The hand detection algorithm supports the application scenarios of air gesture control and the input of the hand key point algorithm. At present, the hand detection algorithm frameworks used are all based on data-driven deep learning algorithms. The algorithm effect is affected by the deep learning model architecture, data enhancement method, and loss function on the one hand, and the data plays a decisive role in the effect of the algorithm itself on the other hand. In the continuous iteration process, the deep learning model will nonlinearly approximate the true value, and the accuracy of the true value of the position where the hand appears in the image directly affects the training effect of the model. In addition to adjusting the hand detection algorithm model itself The performance of the model can be improved by adjusting the model architecture, data enhancement, and other methods. Adding training data and test data to the training model can also improve the performance of the model.
[0004] However, the hand detection algorithm framework currently used may cause the hand detection frame to shake due to false hand detection and missed hand detection, and the shaking of the hand detection frame on the time-series screen will increase the shaking of the hand key point detection algorithm on the time-series screen, which will bring great errors to the results of the hand key point detection algorithm. For the hand detection algorithm, not only the accuracy and recall rate are required, but also the detection results in the actual scene must be stable and smooth enough to effectively support the overall gesture air operation process. Therefore, it is urgent to provide a data annotation solution for hand detection to reduce the shaking of the hand detection frame and improve the stability of the hand detection frame. Summary of the invention
[0005] At least one embodiment of the present disclosure provides a data annotation method, apparatus, device, and storage medium for hand detection.
[0006] In a first aspect, an embodiment of the present disclosure provides a data labeling method for hand detection, the method comprising:
[0007] Get the image containing the gesture;
[0008] Performing hand detection on the image based on at least two hand detection models respectively, and obtaining detection results of each hand detection model, wherein the detection results include: a position of a hand detection frame and an area of the hand detection frame;
[0009] If the number of hand detection frames in the detection results of each hand detection model is the same, then based on the position of the hand detection frame and the area of the hand detection frame in the detection result of each hand detection model, the hand detection frame intersection-over-union ratio between different hand detection models is calculated, and whether the detection result of each hand detection model is qualified is determined based on the hand detection frame intersection-over-union ratio;
[0010] If it is determined to be qualified, the detection results of each hand detection model for the same hand are merged to obtain the detection result of the image as the annotation data of the image.
[0011] In some embodiments, after obtaining the detection result of the image as the annotation data of the image, the data annotation method for hand detection further includes:
[0012] Determine time-series image data; the time-series image data includes: the detection result corresponding to each image within a preset time period;
[0013] Based on the detection results corresponding to each image in the time series image data, if the number of hand detection frames in the detection results of two consecutive upper and lower frames of images is different, or the intersection-and-union ratio of the hand detection frames of two consecutive upper and lower frames of images is less than the preset first intersection-and-union ratio threshold, the time series image data is submitted for manual adjustment.
[0014] In some embodiments, determining whether the detection result of each hand detection model is qualified based on the hand detection frame intersection-over-union ratio includes:
[0015] If the IoU of the hand detection frame is greater than or equal to a preset second IoU threshold, then the detection result of each hand detection model is determined to be qualified;
[0016] If the IoU of the hand detection frame is less than the second IoU threshold, it is determined that the detection result of each hand detection model is unqualified.
[0017] In some embodiments, the detection results of each hand detection model for the same hand are merged to obtain the detection results of the image as the annotation data of the image, including:
[0018] The average value of the positions of the hand detection frames corresponding to the same hand in the detection results of each hand detection model is used as the position of the hand detection frame in the detection results of the image;
[0019] The average value of the area of the hand detection box corresponding to the same hand in the detection result of each hand detection model is used as the area of the hand detection box in the detection result of the image.
[0020] In some embodiments, after obtaining the detection result of each hand detection model, the data labeling method for hand detection further includes:
[0021] If the number of hand detection frames in the detection results of each hand detection model is different, it is determined that the detection result of each hand detection model is unqualified.
[0022] In some embodiments, after determining whether the detection result of each hand detection model is qualified based on the hand detection frame intersection-over-union ratio, the data labeling method for hand detection further includes:
[0023] If it is determined that the detection result of each hand detection model is qualified, after obtaining the detection result of the image as the annotation data of the image, a preset number of data are extracted from the annotation data of the image and submitted for manual review.
[0024] In some embodiments, after obtaining the detection result of each hand detection model, the data labeling method for hand detection further includes:
[0025] If it is determined that the detection result of each hand detection model is unqualified, the detection result of each hand detection model is submitted for manual adjustment.
[0026] In some embodiments, based on the position of the hand detection frame and the area of the hand detection frame in the detection result of each hand detection model, calculating the hand detection frame intersection-over-union ratio between different hand detection models includes:
[0027] For any hand detection model, based on the position of the hand detection frame and the area of the hand detection frame in the detection result of the hand detection model, calculate the area union of the hand detection frame in the hand detection model;
[0028] Calculate the intersection between the area unions of the hand detection boxes in different hand detection models, and the union between the area unions of the hand detection boxes in different hand detection models;
[0029] Based on the intersection between the area unions of the hand detection frames in different hand detection models and the union between the area unions of the hand detection frames in different hand detection models, the intersection-over-union ratio of the hand detection frames between the different hand detection models is calculated.
[0030] In a second aspect, the present disclosure also provides a data annotation device for hand detection, the device comprising:
[0031] An acquisition unit, used for acquiring an image containing a gesture;
[0032] A hand detection unit, configured to perform hand detection on the image based on at least two hand detection models, respectively, to obtain a detection result of each hand detection model, wherein the detection result includes: a position of a hand detection frame and an area of the hand detection frame;
[0033] a determination unit, configured to calculate, if the number of hand detection frames in the detection results of each hand detection model is the same, hand detection frame intersection-over-union ratios between different hand detection models based on the positions of the hand detection frames in the detection results of each hand detection model and the areas of the hand detection frames, and determine whether the detection result of each hand detection model is qualified based on the hand detection frame intersection-over-union ratios;
[0034] The merging unit is used to merge the detection results of each hand detection model on the same hand if the determination unit determines that the hand is qualified, so as to obtain the detection result of the image as the annotation data of the image.
[0035] In a third aspect, an embodiment of the present disclosure further proposes an electronic device, which includes a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the steps of the data labeling method for hand detection as described in the first aspect.
[0036] In a fourth aspect, an embodiment of the present disclosure further proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a program or instructions, and the program or instructions enable a computer to execute the steps of the data labeling method for hand detection as described in the first aspect.
[0037] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, wherein the computer program product includes a computer program, the computer program is stored in a computer-readable storage medium, and at least one processor of a computer reads and executes the computer program from the computer-readable storage medium, so that the computer performs the steps of the data labeling method for hand detection as described in the first aspect.
[0038] It can be seen that in at least one embodiment of the present disclosure, by using multiple models to perform hand detection on an image respectively, the detection results of each hand detection model can be obtained; then by comparing these detection results, if the number of hand detection frames in the detection results of each hand detection model is the same, then based on the position of the hand detection frame in the detection results of each hand detection model and the area of the hand detection frame, it is determined whether the detection result of each hand detection model is qualified; if it is determined to be qualified, it means that the hand detection of the image by the multiple models is close to the true value, then the detection results of the same hand detected by each hand detection model are merged, and the detection results of each hand detection model are comprehensively considered to obtain the detection result of the image, which represents the true situation of the hand in the image. Therefore, using the merged detection result of the image as the annotation data of the image can reduce the annotation error, thereby reducing the jitter of the hand detection frame and improving the stability of the hand detection frame. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure, and a person skilled in the art can also obtain other drawings based on these drawings.
[0040] Figure 1 A flowchart of a data labeling method for hand detection provided by an embodiment of the present disclosure;
[0041] Figure 2 A schematic diagram of a data annotation device for hand detection provided by an embodiment of the present disclosure;
[0042] Figure 3 An exemplary block diagram of an electronic device provided in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION
[0043] In order to more clearly understand the above-mentioned purposes, features and advantages of the present disclosure, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. The specific embodiments described herein are only used to explain the present disclosure, rather than to limit the present disclosure. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in the field belong to the scope of protection of the present disclosure.
[0044] It should be noted that, in this document, relational terms such as “first” and “second” are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.
[0045] The disclosed embodiments provide a data labeling method, device, equipment or storage medium for hand detection, which can quickly, cheaply and efficiently label gesture data for real desensitized disk data, and only requires simple manual discrimination and fine-tuning to semi-automatically generate real labels, thus avoiding the problems of long cycle, high cost, manual error and inability to maintain sufficient stability on the time series screen caused by manual labeling. Thereby, real and diverse gesture data is provided for the training of the hand detection model. In addition, the disclosed embodiments also improve the accuracy and recall rate of the hand detection model by means of secondary verification of the time series information, helping to generate more accurate training data and test data.
[0046] Figure 1 A flow chart of a data annotation method for hand detection provided in an embodiment of the present disclosure is provided. The executor of the data annotation method is an electronic device, and the electronic device includes but is not limited to vehicle-mounted equipment, smart phones, PDAs, tablet computers, wearable devices with display screens, desktop computers, laptop computers, all-in-one machines, smart home devices, servers, etc., wherein the server can be an independent server or a cluster of multiple servers, including servers built locally and servers set up in the cloud.
[0047] The data annotation method for hand detection disclosed in this embodiment belongs to the training process of the hand dynamic detection model. The data annotated by the data annotation method for hand detection is used as part of the training samples for training the dynamic detection model. How to use the annotated data to train the hand dynamic detection model is described below. After the training is completed, the hand dynamic detection model is put into practical use, for example, deployed on the vehicle side, so that the image of the gesture made by the rear passenger during the driving process of the vehicle can be dynamically detected to obtain the position of the hand in the image, so as to facilitate the detection of the position of the hand key point through the hand key point detection algorithm, and then judge the dynamic gesture.
[0048] like Figure 1 As shown, the data annotation method for hand detection may include but is not limited to steps 101 to 104:
[0049] In step 101, an image containing a gesture is acquired.
[0050] In this embodiment, the image containing the gesture comes from the user data that has been stored, wherein the user data that has been stored includes: the image data of the gestures made by the rear passengers of the vehicle during the driving process of the vehicle, which is real user data. In other embodiments, the user data that has been stored can be image data containing user gestures collected in other gesture control scenarios.
[0051] In this embodiment, since the user data that has been written to the disk contains user information, in order to protect user privacy, the user data that has been written to the disk needs to be desensitized to remove sensitive information related to the user. The desensitization processing method is a conventional technical means in this field and will not be described in detail to avoid repetition.
[0052] In step 102, hand detection is performed on the image based on at least two hand detection models to obtain detection results of each hand detection model, where the detection results include: the position of the hand detection frame and the area of the hand detection frame.
[0053] In this embodiment, at least two hand detection models are pre-trained. These hand detection models are only used in the data annotation stage to perform hand detection on the images containing gestures obtained in step 101. For example, two hand detection models (referred to as model A and model B) are pre-trained to perform hand detection on a single image, respectively, to obtain two detection results, which include: the position of the hand detection frame and the area of the hand detection frame. Among them, model A is a hand detection model run by the GPU (Graphics Processing Unit) of the control version end running model, which is superior to the model performance of the vehicle end in terms of speed and accuracy. Model B is a hand detection model with improved Backbone (backbone network) and input image resolution. The number of parameters of model B is much larger than that of model A, and in theory, more accurate hand detection results can be obtained.
[0054] In step 103, if the number of hand detection frames in the detection results of each hand detection model is the same, the hand detection frame intersection-in-union ratio between different hand detection models is calculated based on the position of the hand detection frame and the area of the hand detection frame in the detection results of each hand detection model, and whether the detection result of each hand detection model is qualified is determined based on the hand detection frame intersection-in-union ratio.
[0055] In this embodiment, based on the position of the hand detection frame and the area of the hand detection frame in the detection result of each hand detection model, the intersection over union (IoU) of the hand detection frames between different hand detection models is calculated, specifically:
[0056] For any hand detection model, based on the position of the hand detection frame and the area of the hand detection frame in the detection result of the hand detection model, calculate the area union of the hand detection frame in the hand detection model;
[0057] Calculate the intersection between the area unions of the hand detection boxes in different hand detection models, and the union between the area unions of the hand detection boxes in different hand detection models;
[0058] Based on the intersection between the area unions of the hand detection frames in different hand detection models and the union between the area unions of the hand detection frames in different hand detection models, the intersection-over-union ratio of the hand detection frames between the different hand detection models is calculated.
[0059] For example, there are two hand detection models A and B. The detection result of A includes three hand detection frames (the areas occupied by each are denoted as A1, A2 and A3), and the detection result of B also includes three hand detection frames (the areas occupied by each are denoted as B1, B2 and B3). The union of the areas occupied by the three hand detection frames of A is denoted as the first area (the union of A1, A2 and A3) and the union of the areas occupied by the three hand detection frames of B is denoted as the second area (the union of B1, B2 and B3). The intersection-over-union ratio between the first area and the second area is calculated as the intersection-over-union ratio of the hand detection frames between A and B.
[0060] In this embodiment, whether the detection result of each hand detection model is qualified is determined based on the hand detection frame intersection and union ratio, specifically:
[0061] If the intersection-and-union ratio of the hand detection frame is greater than or equal to a preset second intersection-and-union ratio threshold (for example, 95%), the detection result of each hand detection model is determined to be qualified; if the intersection-and-union ratio of the hand detection frame is less than the first intersection-and-union ratio threshold, the detection result of each hand detection model is determined to be unqualified.
[0062] In some embodiments, if the detection result of each hand detection model is determined to be qualified, after the detection result of the image is obtained as the annotated data of the image in step 104, a preset amount of data is extracted from the annotated data of the image and submitted to manual review to determine the accuracy of the result. The preset amount is, for example, 10%, that is, 10% of the data is extracted from the annotated data of the image and submitted to manual review.
[0063] In some embodiments, after obtaining the detection result of each hand detection model in step 102, if the number of hand detection frames in the detection result of each hand detection model is different, it is directly determined that the detection result of each hand detection model is unqualified.
[0064] In some embodiments, if it is determined that the detection result of each hand detection model is unqualified, indicating that the performances of these detection models diverge, the detection result of each hand detection model is submitted for manual adjustment.
[0065] In step 104, if it is determined to be qualified, the detection results of each hand detection model for the same hand are merged to obtain the detection results of the image as the annotation data of the image.
[0066] In this embodiment, if it is determined that the detection results of each hand detection model are qualified, it means that the detection results of these hand detection models are close to the true value and can be used as labeled data.
[0067] Considering that the detection results of these hand detection models are not exactly the same, the detection results of each hand detection model are combined to obtain the detection results of the image as the annotation data of the image. The detection result of the image comprehensively considers the detection results of different hand detection models, can represent the true value, and reflect the real situation. Therefore, the detection result of the image is used as the annotation data of the image obtained in step 101. The annotation data includes: the position of the hand detection frame in the image and the area of the hand detection frame.
[0068] In some embodiments, an implementation of "combining the detection results of each hand detection model for the same hand to obtain the detection results of the image as the annotation data of the image" in step 104 is:
[0069] The average value of the position of the hand detection box corresponding to the same hand in the detection results of each hand detection model is used as the position of the hand detection box in the detection results of the image; the average value of the area of the hand detection box corresponding to the same hand in the detection results of each hand detection model is used as the area of the hand detection box in the detection results of the image.
[0070] For example, there are two hand detection models A and B, A detects three hand detection frames (denoted as X1, Y1, and Z1), and B detects three hand detection frames (denoted as X2, Y2, and Z2), and X1 matches X2, Y1 matches Y2, and Z1 matches Z2. The matching relationship can be determined based on the position and area of the hand detection frame. For example, if the positions of X1 and X2 are closest and the areas are most similar, then X1 matches X2, that is, X1 and X2 correspond to the same hand.
[0071] Therefore, the average value of the positions of X1 and X2 is used as the position of a hand detection frame in the detection results of the image, the average value of the positions of Y1 and Y2 is used as the position of another hand detection frame in the detection results of the image, and the average value of the positions of Z1 and Z2 is used as the position of yet another hand detection frame in the detection results of the image.
[0072] Therefore, the average value of the areas of X1 and X2 is used as the area of a hand detection frame in the detection results of the image, the average value of the areas of Y1 and Y2 is used as the area of another hand detection frame in the detection results of the image, and the average value of the areas of Z1 and Z2 is used as the area of yet another hand detection frame in the detection results of the image.
[0073] In some embodiments, Figure 1After "determining whether the detection result of each hand detection model is qualified based on the hand detection frame intersection and union ratio" in step 103, the data labeling method for hand detection also includes the following steps:
[0074] If it is determined that the detection results of each hand detection model are qualified, after obtaining the detection results of the image as the annotation data of the image, a preset number of data is extracted from the annotation data of the image and submitted for manual review, where the preset number is, for example, 10%, that is, 10% of the data is extracted from the annotation data of the image and submitted for manual review.
[0075] In summary, by comparing the number of hand detection frames, the position of the hand detection frames, and the area of the hand detection frames detected by model A and model B, the IOU of the corresponding area is determined based on the position of the hand detection frames and the area of the hand detection frames. Data with the same number of hand detection frames and an IOU of 95% or above are treated as qualified, and the detection results of each hand detection model are merged to obtain the detection results of the image as the image annotation data, and 10% of the qualified data are randomly checked. Data with different numbers of hand detection frames or the same number of hand detection frames but IOU less than 95% are treated as unqualified, and manual adjustments are performed, and finally the manually adjusted data is used as the image annotation data.
[0076] It can be seen that in the above embodiments, for the unlabeled images of the disk, by using multiple models to perform hand detection on the images respectively, the detection results of each hand detection model can be obtained; then by comparing these detection results, if the number of hand detection frames in the detection results of each hand detection model is the same, then based on the position of the hand detection frame in the detection results of each hand detection model and the area of the hand detection frame, it is determined whether the detection result of each hand detection model is qualified; if it is determined to be qualified, it means that the hand detection of the image by the multiple models is close to the true value, then the detection results of the same hand detected by each hand detection model are merged, and the detection results of each hand detection model are comprehensively considered to obtain the detection result of the image, which represents the true situation of the hand in the image. Therefore, using the detection result of the merged image as the annotation data of the image can reduce the annotation error, thereby reducing the jitter of the hand detection frame and improving the stability of the hand detection frame.
[0077] Based on manual labeling, the embodiments of the present disclosure can expand the training data set. At the same time, the labeled data obtained by the embodiments of the present disclosure is more stable than manual labeling, and is better than manual labeling to a certain extent. While expanding the scene data set, it saves time and energy for manual labeling, and is faster and more efficient.
[0078] Based on the above embodiments, Figure 1After "obtaining the detection result of the image as the annotation data of the image" in step 104, the data annotation method for hand detection may further include the following steps:
[0079] Determine the time series image data; the time series image data includes: the detection results corresponding to each image within a preset time period, wherein the detection results are the detection results of the annotation data of the corresponding image. Based on the detection results corresponding to each image in the time series image data, if the number of hand detection frames in the detection results of two consecutive upper and lower frames of images is different, or the intersection-and-union ratio of the hand detection frames of the two consecutive upper and lower frames of images is less than the preset first intersection-and-union ratio threshold (for example, 20%), it means that the performance of the hand detection model in the two consecutive upper and lower frames of images has diverged, and the time series image data is submitted for manual adjustment. Among them, the first intersection-and-union ratio threshold is less than the aforementioned second intersection-and-union ratio threshold.
[0080] In the disclosed embodiment, when the hand detection model shows divergence in the performance of two consecutive upper and lower frames of images, the time-series image data is submitted for manual adjustment to implement time-series self-checking, which can effectively obtain missed or misdetected data.
[0081] Moreover, since manual frame-by-frame labeling has more messy errors than model labeling, especially for scenes with continuous images, it is impossible to obtain an effective and stable detection frame without manually labeling the upper and lower frames, and jitter will definitely occur, and this jitter is not within the controllable range. This is because it is impossible for labelers to meet clear indicators and require fine pixel-level labeling of the hand area. For such labeling tasks, the disclosed embodiments will reduce the jitter errors caused by manual labeling and improve the performance stability of the model on continuous frames. For the overall task, this can improve the refinement of the scene and improve the user's operability.
[0082] It can be seen that the embodiment of the present disclosure includes an automated process (i.e., the process of the aforementioned data labeling method for hand detection) and a manual review process. Therefore, the process of this embodiment can be understood as a semi-automated process, not a fully automated process.
[0083] Based on the above embodiment, the data labeling method for hand detection further includes the following steps:
[0084] The images with labeled data are used as training sample data for training the hand dynamic detection model. When training the hand dynamic detection model, the images with labeled data are input into the machine learning model, and the parameters of the machine learning model itself are adjusted until the detection result of the image output by the machine learning model is the labeled data of the image or converges to the labeled data, and the training is completed to obtain the hand dynamic detection model. It should be noted that the specific process of training the hand dynamic detection model follows the training process in the field of neural networks and will not be repeated here.
[0085] Among them, the image with labeled data is Figure 1 The labeled data mentioned in step 104 is annotated on the image to obtain an image with labeled data. The image with labeled data introduces manually reviewed labeled data to achieve fine-tuning of the labeled data. This part of data is for scenes with poor performance compared to the online model. Adding this batch of data for training can effectively improve the accuracy and recall of the online model. In scenarios where the initial machine learning model may cause false hand detection and missed hand detection due to seat belt positions, toys, feet, and wrinkled clothes, the initial machine learning model is trained by introducing images of manually reviewed labeled data, so that the hand dynamic detection model obtained after training reduces the cases of missed hand detection and false hand detection, effectively improving the accuracy and stability of remote operation.
[0086] In addition, this embodiment can quickly establish training data sets in a variety of scenarios, which can reduce the cost of manual labeling while quickly shortening the time, reduce the errors caused by manual labeling, make the algorithm more robust, and improve performance in different scenarios.
[0087] Based on the above embodiment, verification was performed on 450,000 sets of data, 370,000 sets of data were quickly annotated, and 70,000 sets of data were manually fine-tuned. Compared with manually annotated data, the cost is saved by more than 90% and the speed is increased by more than 80%. In terms of data validity, adding this batch of data for training can improve the accuracy and recall rate of the model. On the time series screen, the model performance is more stable and the detection frame is more stable.
[0088] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all described as a series of action combinations, but those skilled in the art can understand that the embodiments of the present disclosure are not limited by the described action sequence, because according to the embodiments of the present disclosure, some steps can be performed in other sequences or simultaneously. In addition, those skilled in the art can understand that the embodiments described in the specification are all optional embodiments.
[0089] Figure 2 The present invention provides a schematic diagram of a data annotation device for hand detection provided in an embodiment of the present invention. The data annotation device can be applied to electronic devices, including but not limited to vehicle-mounted devices, smart phones, PDAs, tablet computers, wearable devices with display screens, desktop computers, laptop computers, all-in-one machines, smart home devices, servers, etc., wherein the server can be an independent server or a cluster of multiple servers, including servers built locally and servers built in the cloud. The data annotation device for hand detection provided in an embodiment of the present invention can execute the processing flow provided in each embodiment of the data annotation method for hand detection, such as Figure 2As shown, the data annotation device for hand detection includes but is not limited to: an acquisition unit 21, a hand detection unit 22, a determination unit 23 and a merging unit 24. The functions of each unit are described as follows:
[0090] An acquisition unit 21, used to acquire an image containing a gesture;
[0091] A hand detection unit 22, configured to perform hand detection on the image based on at least two hand detection models, respectively, to obtain a detection result of each hand detection model, wherein the detection result includes: a position of a hand detection frame and an area of the hand detection frame;
[0092] a determination unit 23, configured to calculate, if the number of hand detection frames in the detection results of each hand detection model is the same, hand detection frame intersection-over-union ratios between different hand detection models based on the positions of the hand detection frames and the areas of the hand detection frames in the detection results of each hand detection model, and determine whether the detection result of each hand detection model is qualified based on the hand detection frame intersection-over-union ratio;
[0093] The merging unit 24 is used to merge the detection results of each hand detection model on the same hand if the determination unit 23 determines that it is qualified, so as to obtain the detection result of the image as the annotation data of the image.
[0094] In some embodiments, the determination unit 23 is further used to determine the time series image data after the merging unit 24 obtains the detection result of the image as the annotation data of the image; the time series image data includes: the detection result corresponding to each image in a preset time period. The data annotation device for hand detection also includes a submission unit for submitting the time series image data for manual adjustment based on the detection result corresponding to each image in the time series image data, if the number of hand detection frames in the detection results of the upper and lower consecutive frames of images is different, or the intersection-and-union ratio of the hand detection frames of the upper and lower consecutive frames of images is less than a preset first intersection-and-union ratio threshold.
[0095] In some embodiments, the determination unit 23 determines whether the detection result of each hand detection model is qualified based on the hand detection frame intersection and union ratio, including:
[0096] If the IoU of the hand detection frame is greater than or equal to a preset second IoU threshold, then the detection result of each hand detection model is determined to be qualified;
[0097] If the IoU of the hand detection frame is less than the second IoU threshold, it is determined that the detection result of each hand detection model is unqualified.
[0098] In some embodiments, the merging unit 24 merges the detection results of each hand detection model for the same hand to obtain the detection results of the image as the annotation data of the image, including:
[0099] The average value of the positions of the hand detection frames corresponding to the same hand in the detection results of each hand detection model is used as the position of the hand detection frame in the detection results of the image;
[0100] The average value of the area of the hand detection box corresponding to the same hand in the detection result of each hand detection model is used as the area of the hand detection box in the detection result of the image.
[0101] In some embodiments, the determination unit 23 is further configured to determine that the detection result of each hand detection model is unqualified if the number of hand detection frames in the detection result of each hand detection model is different.
[0102] In some embodiments, the submission unit is further used to extract a preset amount of data from the annotated data of the image and submit it for manual review after obtaining the detection result of the image as the annotation data of the image if it is determined that the detection result of each hand detection model is qualified.
[0103] In some embodiments, the submitting unit is further configured to submit the detection result of each hand detection model for manual adjustment if it is determined that the detection result of each hand detection model is unqualified.
[0104] In some embodiments, the determination unit 23 calculates the hand detection frame intersection-over-union ratio between different hand detection models based on the position of the hand detection frame and the area of the hand detection frame in the detection result of each hand detection model, including:
[0105] For any hand detection model, based on the position of the hand detection frame and the area of the hand detection frame in the detection result of the hand detection model, calculate the area union of the hand detection frame in the hand detection model;
[0106] Calculate the intersection between the area unions of the hand detection boxes in different hand detection models, and the union between the area unions of the hand detection boxes in different hand detection models;
[0107] Based on the intersection between the area unions of the hand detection frames in different hand detection models and the union between the area unions of the hand detection frames in different hand detection models, the intersection-over-union ratio of the hand detection frames between the different hand detection models is calculated.
[0108] It can be seen that in at least one data labeling device embodiment of the present disclosure, by using multiple models to perform hand detection on images respectively, the detection results of each hand detection model can be obtained; then by comparing these detection results, if the number of hand detection frames in the detection results of each hand detection model is the same, then based on the position of the hand detection frame in the detection results of each hand detection model and the area of the hand detection frame, it is determined whether the detection result of each hand detection model is qualified; if it is determined to be qualified, it means that the hand detection of the image by the multiple models is close to the true value, then the detection results of the same hand detected by each hand detection model are merged, and the detection results of each hand detection model are comprehensively considered to obtain the detection result of the image, which represents the true situation of the hand in the image. Therefore, using the merged detection result of the image as the annotation data of the image can reduce the annotation error, thereby reducing the jitter of the hand detection frame and improving the stability of the hand detection frame.
[0109] Figure 3 is an exemplary block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 3 As shown, the electronic device includes: a memory 31, a processor 32, and a computer program stored in the memory 31. It can be understood that the memory 31 in this embodiment can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories.
[0110] In some implementations, the memory 31 stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system and application programs.
[0111] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic tasks and process hardware-based tasks. The application includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application tasks. The program that implements the data annotation method for hand detection provided by the embodiment of the present disclosure can be included in the application.
[0112] In the embodiment of the present disclosure, at least one processor 32 calls a program or instruction stored in at least one memory 31, specifically, a program or instruction stored in an application, and at least one processor 32 is used to execute the steps of each embodiment of the data labeling method for hand detection provided in the embodiment of the present disclosure.
[0113] The data annotation method for hand detection provided by the embodiment of the present disclosure can be applied to the processor 32, or implemented by the processor 32. The processor 32 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 32 or the instructions in the form of software. The above-mentioned processor 32 can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.
[0114] The steps of the data annotation method for hand detection provided by the embodiment of the present disclosure can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 31, and the processor 32 reads the information in the memory 31 and completes the steps of the method in combination with its hardware.
[0115] The embodiments of the present disclosure also provide a computer-readable storage medium, wherein the computer-readable storage medium stores a program or instruction, wherein the program or instruction enables a computer to execute the steps of each embodiment of the data labeling method for hand detection.
[0116] The present disclosure also provides a computer program product, which includes a computer program stored in a computer-readable storage medium, which may be a non-transitory computer-readable storage medium. At least one processor of a computer reads and executes the computer program from the computer-readable storage medium, so that the computer performs the steps of each embodiment of the data labeling method for hand detection.
[0117] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0118] Those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present disclosure and to form different embodiments.
[0119] Those skilled in the art will appreciate that the description of each embodiment has its own emphasis, and for parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0120] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A data annotation method for hand detection, the method comprising: Get the image containing the gesture; Performing hand detection on the image based on at least two hand detection models respectively, and obtaining detection results of each hand detection model, wherein the detection results include: a position of a hand detection frame and an area of the hand detection frame; If the number of hand detection frames in the detection results of each of the hand detection models is the same, calculating the hand detection frame intersection-over-union ratios between different hand detection models based on the positions of the hand detection frames and the areas of the hand detection frames in the detection results of each of the hand detection models, and determining whether the detection result of each of the hand detection models is qualified based on the hand detection frame intersection-over-union ratios; If it is determined to be qualified, the detection results of each hand detection model for detecting the same hand are merged to obtain the detection result of the image as the annotation data of the image.
2. The method according to claim 1, wherein: After obtaining the detection result of the image as the annotation data of the image, the method further includes: Determine time-series image data; the time-series image data includes: the detection result corresponding to each image within a preset time period; Based on the detection results corresponding to each image in the time-series image data, if the number of hand detection frames in the detection results of two consecutive upper and lower frames of images is different, or the intersection-and-union ratio of the hand detection frames of two consecutive upper and lower frames of images is less than a preset first intersection-and-union ratio threshold, the time-series image data is submitted for manual adjustment.
3. The method according to claim 1, wherein: The determining whether the detection result of each hand detection model is qualified based on the hand detection frame intersection-over-union ratio includes: If the IoU of the hand detection frame is greater than or equal to a preset second IoU threshold, determining that the detection result of each hand detection model is qualified; If the IoU of the hand detection frame is less than the second IoU threshold, it is determined that the detection result of each hand detection model is unqualified.
4. The method according to claim 1, wherein: The step of merging the detection results of each hand detection model for detecting the same hand to obtain the detection result of the image as the annotation data of the image includes: taking the average value of the positions of the hand detection frames corresponding to the same hand in the detection results of each of the hand detection models as the position of the hand detection frame in the detection results of the image; The average value of the area of the hand detection frame corresponding to the same hand in the detection results of each of the hand detection models is used as the area of the hand detection frame in the detection results of the image.
5. The method according to claim 1, wherein: After obtaining the detection result of each hand detection model, the method further includes: If the number of hand detection frames in the detection results of each of the hand detection models is different, it is determined that the detection result of each of the hand detection models is unqualified.
6. The method according to claim 1, wherein: After determining whether the detection result of each hand detection model is qualified based on the hand detection frame intersection-over-union ratio, the method further includes: If it is determined that the detection result of each of the hand detection models is qualified, after obtaining the detection result of the image as the annotation data of the image, a preset number of data are extracted from the annotation data of the image and submitted for manual review.
7. The method according to claim 3 or 5, wherein: After obtaining the detection result of each hand detection model, the method further includes: If it is determined that the detection result of each of the hand detection models is unqualified, the detection result of each of the hand detection models is submitted for manual adjustment.
8. The method according to claim 1, wherein the calculating the hand detection frame intersection-over-union ratio between different hand detection models based on the position of the hand detection frame and the area of the hand detection frame in the detection result of each hand detection model comprises: For any of the hand detection models, based on the position of the hand detection frame and the area of the hand detection frame in the detection result of the hand detection model, calculate the area union of the hand detection frame in the hand detection model; Calculate the intersection between the area unions of the hand detection boxes in different hand detection models, and the union between the area unions of the hand detection boxes in different hand detection models; Based on the intersection between the area unions of the hand detection frames in different hand detection models and the union between the area unions of the hand detection frames in different hand detection models, the intersection-over-union ratio of the hand detection frames between the different hand detection models is calculated.
9. A data annotation device for hand detection, the device comprising: An acquisition unit, used for acquiring an image containing a gesture; A hand detection unit, configured to perform hand detection on the image based on at least two hand detection models, respectively, to obtain a detection result of each hand detection model, wherein the detection result includes: a position of a hand detection frame and an area of the hand detection frame; a determination unit, configured to calculate, if the number of hand detection frames in the detection results of each of the hand detection models is the same, hand detection frame intersection-and-union ratios between different hand detection models based on the positions of the hand detection frames and the areas of the hand detection frames in the detection results of each of the hand detection models, and determine whether the detection result of each of the hand detection models is qualified based on the hand detection frame intersection-and-union ratios; A merging unit is used to merge the detection results of each hand detection model for detecting the same hand if the determination unit determines that the hand is qualified, so as to obtain the detection result of the image as the annotation data of the image.
10. An electronic device, wherein: The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the data labeling method for hand detection according to any one of claims 1 to 8.
11. A computer-readable storage medium, wherein: The computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute the steps of the data labeling method for hand detection as claimed in any one of claims 1 to 8.