Data annotation method and device, equipment and storage medium

Abstract

The invention relates to a data annotation method and device, equipment and a storage medium. In at least one embodiment of the invention, the positions of the hand key points in the image are detected by using the multiple models, so that hand key point sets corresponding to the multiple models can be obtained; the relative position deviation between the hand key point sets is calculated, if the relative position deviation is smaller than or equal to a preset relative position deviation threshold value, it shows that the positions of the hand key points detected by the multiple models are close to the real positions, and the hand key point sets are comprehensively considered to obtain the position information of the hand key points; the position information of each hand key point represents the real position of each hand key point, so that by taking the position information of each hand key point as the annotation data of the image, the annotation error can be reduced, the shake of the hand key point is reduced, and the stability of the hand key point is improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of gesture control technology, and in particular to a data labeling method, device, equipment and storage medium. Background Art

[0002] With the development of vehicle technology, the demand for controlling the rear entertainment screen of the vehicle through gestures has increased. At present, dynamic gestures are judged by detecting the position of key points of the hand, and then the up and down waving, clenching fists, opening fists and other actions are recognized to control the return, click, release and other functions on the screen.

[0003] The performance of the hand key point detection algorithm directly affects the recognition effect of dynamic gestures. The hand key point detection algorithm framework currently used is based on a data-driven deep learning algorithm. The algorithm effect is affected by the deep learning model architecture, data enhancement method, and loss function on the one hand, and on the other hand, data plays a decisive role in the effect of the algorithm itself. In the continuous iteration process, the deep learning model will nonlinearly approximate the true value, and the accuracy of the true value of the hand key point position directly affects the training effect of the model.

[0004] In order to improve the detection effect of the hand key point detection algorithm model, the real values ​​of the hand key points are usually annotated by manual labeling to improve the training effect of the model. However, the cost of manual labeling is high and there may be labeling errors. For example, there are labeling errors in the hand key points manually labeled on the time series screen, which leads to obvious jitter of the hand key points visually found on the time series screen. Using manually labeled data to train the model will cause the model to perform poorly on the time series screen. For example, the cursor point on the screen is unstable when the hand does not move. Summary of the invention

[0005] At least one embodiment of the present disclosure provides a data annotation method, apparatus, device, and storage medium.

[0006] In a first aspect, an embodiment of the present disclosure provides a data labeling method, the method comprising:

[0007] Get the image containing the gesture;

[0008] Detect hand key points of the image based on at least two hand key point detection models respectively, and obtain a hand key point set corresponding to each detection model;

[0009] Calculate the relative position deviation between the hand key point sets corresponding to each detection model;

[0010] If the relative position deviation between the hand key point sets corresponding to each detection model is less than or equal to a preset relative position deviation threshold, the position information of the hand key points in the image is determined based on the hand key point sets corresponding to each detection model as the image annotation data.

[0011] In some embodiments, at least two hand key point detection models respectively adopt one or more different trained models in model architecture, image resolution, input image size and backbone network.

[0012] In some embodiments, before calculating the relative position deviation between the hand key point sets corresponding to each detection model, the data labeling method further includes:

[0013] The hand key point set corresponding to each detection model is subjected to background false detection filtering to filter out the points located on the image background as hand key points.

[0014] In some embodiments, calculating the relative position deviation between the hand key point sets corresponding to each detection model includes:

[0015] Based on the position of each hand key point in the hand key point set corresponding to each detection model, the Euclidean distance between the position of each hand key point in different hand key point sets is calculated;

[0016] The average value of the Euclidean distance of the position of each hand key point in different hand key point sets is calculated, and the average Euclidean distance is obtained as the relative position deviation.

[0017] In some embodiments, if the relative position deviation between the hand key point sets corresponding to each detection model is less than or equal to a preset relative position deviation threshold, determining the position information of the hand key points in the image as the annotation data of the image based on the hand key point sets corresponding to each detection model includes:

[0018] If the average Euclidean distance is less than or equal to the preset average Euclidean distance threshold, the average value of the position of each hand key point in different hand key point sets is used as the position information of each hand key point, and the position information of each hand key point is used as the annotation data of the image.

[0019] In some embodiments, after obtaining the average Euclidean distance, the data labeling method further includes:

[0020] If the average Euclidean distance is greater than a preset average Euclidean distance threshold, the hand key point set corresponding to each detection model is associated and stored in a file to be manually reviewed.

[0021] In some embodiments, after determining the position information of the hand key points in the image as the annotation data of the image based on the hand key point set corresponding to each detection model, the data annotation method further includes:

[0022] The image where the key points of the hand as the annotation data are located is used as the associated image of the annotation data;

[0023] When training the dynamic detection model of the key points of the hand, the associated image of the annotated data is input into the machine learning model, and the parameters of the machine learning model itself are adjusted until the position output by the machine learning model is the position in the annotated data or converges to the position in the annotated data. The training is completed to obtain the dynamic detection model of the key points of the hand.

[0024] In a second aspect, the present disclosure also provides a data labeling device, the device comprising:

[0025] An acquisition unit, used for acquiring an image containing a gesture;

[0026] A detection unit, configured to detect hand key points of an image based on at least two detection models of hand key points, and obtain a set of hand key points corresponding to each detection model;

[0027] A calculation unit, used for calculating the relative position deviation between the hand key point sets corresponding to each detection model;

[0028] A determination unit is used to determine the position information of the hand key points in the image as the annotation data of the image based on the hand key point sets corresponding to each detection model if the relative position deviation between the hand key point sets corresponding to each detection model is less than or equal to a preset relative position deviation threshold.

[0029] In a third aspect, an embodiment of the present disclosure further proposes an electronic device, which includes a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the steps of the data labeling method described in the first aspect.

[0030] In a fourth aspect, an embodiment of the present disclosure further proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a program or instructions, and the program or instructions enable a computer to execute the steps of the data labeling method described in the first aspect.

[0031] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, wherein the computer program product includes a computer program, the computer program is stored in a computer-readable storage medium, and at least one processor of a computer reads and executes the computer program from the computer-readable storage medium, so that the computer performs the steps of the data labeling method described in the first aspect.

[0032] It can be seen that in at least one embodiment of the present disclosure, by using multiple models to detect the positions of hand key points in the image, a set of hand key points corresponding to each of the multiple models can be obtained; then, by calculating the relative position deviation between these hand key point sets, if the relative position deviation is less than or equal to the preset relative position deviation threshold, it means that the position of the hand key points detected by the multiple models is close to the true position, then the position information of each hand key point is obtained by comprehensively considering these hand key point sets, which represents the true position of each hand key point. Therefore, using the position information of each hand key point as the annotation data of the image can reduce the annotation error, thereby reducing the jitter of the hand key points and improving the stability of the hand key points. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure, and a person skilled in the art can also obtain other drawings based on these drawings.

[0034] Figure 1 A flowchart of a data labeling method provided by an embodiment of the present disclosure;

[0035] Figure 2 A schematic diagram of a data labeling device provided in an embodiment of the present disclosure;

[0036] Figure 3 An exemplary block diagram of an electronic device provided in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] In order to more clearly understand the above-mentioned purposes, features and advantages of the present disclosure, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. The specific embodiments described herein are only used to explain the present disclosure, rather than to limit the present disclosure. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in the field belong to the scope of protection of the present disclosure.

[0038] It should be noted that, in this document, relational terms such as “first” and “second” are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0039] In order to recognize gestures, hand key points are pre-set, for example, 21 key points are set on the hand, and these 21 key points include points at the hand joint positions. Those skilled in the art can set the hand key points according to actual needs, and this embodiment does not limit the number and specific setting method of the hand key points.

[0040] The disclosed embodiments provide a data labeling method, apparatus, device or storage medium. By using multiple models to detect the positions of hand key points in an image, a set of hand key points corresponding to each of the multiple models can be obtained. Then, by calculating the relative position deviation between these sets of hand key points, if the relative position deviation is less than or equal to a preset relative position deviation threshold, it means that the position of the hand key points detected by the multiple models is close to the real position. Then, by comprehensively considering these sets of hand key points, the position information of each hand key point is obtained, which represents the real position of each hand key point. Therefore, using the position information of each hand key point as the labeling data of the image can reduce the labeling error, thereby reducing the jitter of the hand key points and improving the stability of the hand key points.

[0041] Figure 1 A flow chart of a data labeling method provided for an embodiment of the present disclosure, wherein the executor of the data labeling method is an electronic device, and the electronic device includes but is not limited to vehicle-mounted equipment, smart phones, PDAs, tablet computers, wearable devices with display screens, desktop computers, laptop computers, all-in-one machines, smart home devices, servers, etc., wherein the server can be an independent server or a cluster of multiple servers, including servers built locally and servers set up in the cloud.

[0042] The data annotation method disclosed in this embodiment belongs to the training process of the dynamic detection model of the hand key points. The data annotated by the data annotation method is used as part of the training samples for training the dynamic detection model. The specific method of using the annotated data to train the dynamic detection model is described below. Among them, the dynamic detection model of the hand key points is put into practical use after the training is completed. For example, it is deployed on the vehicle side so that the images of the gestures made by the rear passengers can be dynamically detected during the driving process of the vehicle to obtain the hand key points, so as to facilitate the gesture recognition algorithm to use the dynamically detected hand key points to identify specific gestures.

[0043] like Figure 1 As shown, the data annotation method may include but is not limited to steps 101 to 104:

[0044] In step 101, an image containing a gesture is acquired.

[0045] In this embodiment, the image containing the gesture comes from the user data that has been stored, wherein the user data that has been stored includes: the image data of the gestures made by the rear passengers of the vehicle during the driving process of the vehicle, which is real user data. In other embodiments, the user data that has been stored can be image data containing user gestures collected in other gesture control scenarios.

[0046] In this embodiment, since the user data that has been written to the disk contains user information, in order to protect user privacy, the user data that has been written to the disk needs to be desensitized to remove sensitive information related to the user. The desensitization processing method is a conventional technical means in this field and will not be described in detail to avoid repetition.

[0047] In step 102, hand key points of the image are detected based on at least two detection models of hand key points, and a set of hand key points corresponding to each detection model is obtained.

[0048] In this embodiment, at least two detection models of hand key points are pre-trained. These detection models are only used in the data annotation stage to detect hand key points on the image containing the gesture acquired in step 101.

[0049] In some embodiments, at least two detection models of hand key points respectively use one or more different trained models in model architecture, image resolution, input image size and backbone network (BackBone). Compared with the neural network processing model (such as cdsp model or npu model) running on the vehicle side, these detection models run on GPU (Graphics Processing Unit, graphics processor) with better accuracy and speed.

[0050] In this embodiment, the hand key points of the image containing the gesture acquired in step 101 are detected based on at least two detection models of the hand key points, and a hand key point set corresponding to each detection model can be obtained, wherein the hand key point set includes the identifier (ID) of the hand key point and the position of the hand key point in the image. For ease of description, the position of the hand key point in the image is described as the position of the hand key point in the following.

[0051] In step 103, the relative position deviation between the hand key point sets corresponding to each detection model is calculated.

[0052] In this embodiment, the corresponding positions of the same hand key point in different hand key point sets can be compared to obtain the relative position deviation of the same hand key point in different hand key point sets. Then, by comprehensively considering the relative position deviation of each hand key point in different hand key point sets, the relative position deviation between the hand key point sets corresponding to each detection model can be obtained.

[0053] In some embodiments, before calculating the relative position deviation between the hand key point sets corresponding to each detection model, the data labeling method further includes: performing background false detection filtering on the hand key point sets corresponding to each detection model to filter out the points located on the image background as hand key points. Among them, background false detection filtering is a conventional technical means in this field, and will not be described in detail to avoid repetition.

[0054] In step 104, if the relative position deviation between the hand key point sets corresponding to each detection model is less than or equal to a preset relative position deviation threshold, the position information of the hand key points in the image is determined based on the hand key point sets corresponding to each detection model as the annotation data of the image.

[0055] In this embodiment, the relative position deviation between the hand key point sets corresponding to each detection model is less than or equal to a preset relative position deviation threshold, indicating that the positions of the hand key points detected by these detection models are close to the actual positions and can be considered as the actual positions and can be used as annotation data.

[0056] Considering that the positions of the hand key points detected by these detection models are not exactly the same, the position information of the hand key points in the image is determined based on the hand key point set corresponding to each detection model. The position information comprehensively considers the positions of the hand key points in different sets and can represent the real positions of the hand key points. Therefore, the position information is used as the annotation data of the image obtained in step 101. The annotation data includes: the real position of each hand key point and the identifier (ID) of each hand key point.

[0057] It can be seen that in the above embodiments, for the unlabeled image of the disk, by using multiple models (at least two detection models of hand key points) to detect the positions of the hand key points in the image, the hand key point sets corresponding to the multiple models can be obtained; then, by calculating the relative position deviation between these hand key point sets, if the relative position deviation is less than or equal to the preset relative position deviation threshold, it means that the position of the hand key points detected by the multiple models is close to the true position, then the position information of each hand key point is obtained by comprehensively considering these hand key point sets, which represents the true position of each hand key point. Therefore, using the position information of each hand key point as the annotation data of the image can reduce the annotation error, thereby reducing the jitter of the hand key points and improving the stability of the hand key points.

[0058] Based on manual labeling, the embodiments of the present disclosure can expand the training data set. At the same time, the labeled data obtained by the embodiments of the present disclosure is more stable than manual labeling, and is better than manual labeling to a certain extent. While expanding the scene data set, it saves time and energy for manual labeling, and is faster and more efficient.

[0059] Based on the above embodiments, Figure 1 One implementation of step 103 of "calculating the relative position deviation between the hand key point sets corresponding to each detection model" is:

[0060] Based on the position of each hand key point in the hand key point set corresponding to each detection model, the Euclidean distance of the position of each hand key point in different hand key point sets is calculated; the average value of the Euclidean distance of the position of each hand key point in different hand key point sets is calculated to obtain the average Euclidean distance as the relative position deviation.

[0061] Taking three hand key points as an example, the Euclidean distance of the position of hand key point 1 in different sets of hand key points is recorded as Euclidean distance 1, the Euclidean distance of the position of hand key point 2 in different sets of hand key points is recorded as Euclidean distance 2, and the Euclidean distance of the position of hand key point 3 in different sets of hand key points is recorded as Euclidean distance 3. Average Euclidean distance = (Euclidean distance 1 + Euclidean distance 2 + Euclidean distance 3) / 3.

[0062] Based on the above embodiments, Figure 1 One implementation of step 104 of “if the relative position deviation between the hand key point sets corresponding to each detection model is less than or equal to a preset relative position deviation threshold, then determining the position information of the hand key points in the image as the annotation data of the image based on the hand key point sets corresponding to each detection model” is:

[0063] If the average Euclidean distance is less than or equal to the preset average Euclidean distance threshold, the average value of the position of each hand key point in different hand key point sets is used as the position information of each hand key point, and the position information of each hand key point is used as the annotation data of the image.

[0064] For example, the average Euclidean distance threshold is 2 pixels (pixel). If the average value of the Euclidean distances of the positions of each hand key point in different sets of hand key points (i.e., the average Euclidean distance) is less than or equal to 2 pixels, the average value of the positions of each hand key point in different sets of hand key points is used as the position information of each hand key point. For example, taking three hand key points as an example, the average value of the positions of hand key point 1 in different sets of hand key points is used as the position information of hand key point 1; the average value of the positions of hand key point 2 in different sets of hand key points is used as the position information of hand key point 2; and the average value of the positions of hand key point 3 in different sets of hand key points is used as the position information of hand key point 3.

[0065] In some embodiments, after the position information of each key point of the hand is used as the annotation data of the image, part of the annotation data (e.g., 10% of the annotation data) can be selected for manual review to determine the accuracy of the result. It can be seen that this embodiment includes an automated process (i.e., the process of the aforementioned data annotation method) and a manual review process. Therefore, the process of this embodiment can be understood as a semi-automated process, not a fully automated process.

[0066] If the average Euclidean distance is greater than a preset average Euclidean distance threshold, the hand key point set corresponding to each detection model is associated and stored in a file to be manually reviewed.

[0067] For example, if the average Euclidean distance is greater than 2 pixels, it means that the performances of these detection models diverge. Therefore, the hand key point sets corresponding to each detection model are associated and stored in the file to be manually reviewed for manual review and fine-tuning.

[0068] Based on the above embodiments, Figure 1 After "determining the position information of the hand key points in the image as the annotation data of the image based on the hand key point set corresponding to each detection model" in step 104, the data annotation method further includes the following steps:

[0069] The image where the key points of the hand as the annotation data are located is used as the associated image of the annotation data;

[0070] When training the dynamic detection model of the key points of the hand, the associated image of the annotated data is input into the machine learning model, and the parameters of the machine learning model itself are adjusted until the position output by the machine learning model is the position in the annotated data or converges to the position in the annotated data. The training is completed to obtain the dynamic detection model of the key points of the hand.

[0071] In this embodiment, the annotated data and the associated images of the annotated data are used as training samples for training the dynamic detection model of the key points of the hand. Furthermore, when training the dynamic detection model of the key points of the hand, the associated images of the annotated data are input into the machine learning model, and the parameters of the machine learning model itself are adjusted until the position output by the machine learning model is the position in the annotated data or converges to the position in the annotated data, and the training is completed to obtain the dynamic detection model of the key points of the hand. It should be noted that the specific process of training the dynamic detection model follows the training process in the field of neural networks and will not be repeated.

[0072] This embodiment can quickly establish training data sets in a variety of scenarios, which can quickly shorten the time while reducing the cost of manual labeling, making the algorithm more robust and improving performance in different scenarios.

[0073] On the basis of the above embodiment, the stability of the prediction result of the hand key point on the time series screen is represented by a measurement index based on continuous data (for example, two consecutive frames of images) (for example, the movement acceleration between the hand key point of the previous frame image and the hand key point of the next frame image). The data annotated by the above-mentioned annotation method can reduce the annotation error, thereby reducing the jitter of the hand key point, improving the stability of the hand key point, and showing more stable and accurate performance on the time series screen.

[0074] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all described as a series of action combinations, but those skilled in the art can understand that the embodiments of the present disclosure are not limited by the described action sequence, because according to the embodiments of the present disclosure, some steps can be performed in other sequences or simultaneously. In addition, those skilled in the art can understand that the embodiments described in the specification are all optional embodiments.

[0075] Figure 2 The following is a schematic diagram of a data annotation device provided in an embodiment of the present disclosure. The data annotation device can be applied to electronic devices, including but not limited to vehicle-mounted devices, smart phones, PDAs, tablet computers, wearable devices with display screens, desktop computers, laptop computers, all-in-one machines, smart home devices, servers, etc., wherein the server can be an independent server or a cluster of multiple servers, including servers built locally and servers built in the cloud. The data annotation device provided in an embodiment of the present disclosure can execute the processing flow provided in each embodiment of the data annotation method, such as Figure 2 As shown, the data annotation device includes but is not limited to: an acquisition unit 21, a detection unit 22, a calculation unit 23 and a determination unit 24. The functions of each unit are described as follows:

[0076] An acquisition unit 21, used to acquire an image containing a gesture;

[0077] A detection unit 22, configured to detect hand key points of an image based on at least two detection models of hand key points, and obtain a set of hand key points corresponding to each detection model;

[0078] A calculation unit 23, used for calculating the relative position deviation between the hand key point sets corresponding to each detection model;

[0079] The determination unit 24 is used to determine the position information of the hand key points in the image as the annotation data of the image based on the hand key point sets corresponding to each detection model if the relative position deviation between the hand key point sets corresponding to each detection model is less than or equal to a preset relative position deviation threshold.

[0080] In some embodiments, at least two hand key point detection models respectively adopt one or more different trained models in model architecture, image resolution, input image size and backbone network.

[0081] In some embodiments, the data labeling device also includes a filtering unit for performing background false detection filtering on the hand key point set corresponding to each detection model to filter out points on the image background that serve as hand key points.

[0082] In some embodiments, the computing unit 23 is configured to:

[0083] Based on the position of each hand key point in the hand key point set corresponding to each detection model, the Euclidean distance between the position of each hand key point in different hand key point sets is calculated;

[0084] The average value of the Euclidean distance of the position of each hand key point in different hand key point sets is calculated, and the average Euclidean distance is obtained as the relative position deviation.

[0085] In some embodiments, the determination unit 24 is used to take the average value of the position of each hand key point in different hand key point sets as the position information of each hand key point if the average Euclidean distance is less than or equal to a preset average Euclidean distance threshold, and take the position information of each hand key point as the annotation data of the image.

[0086] In some embodiments, the data labeling device also includes a storage unit, which is used to associate and store the hand key point set corresponding to each detection model in the file to be manually reviewed after the calculation unit 23 obtains the average Euclidean distance, if the average Euclidean distance is greater than a preset average Euclidean distance threshold.

[0087] In some embodiments, the data labeling device also includes a training unit, which is used to determine that after the unit 24 uses the position information of each hand key point as the labeling data of the image, the image where the hand key point is located as the labeling data is used as the associated image of the labeling data; when training the dynamic detection model of the hand key points, the associated image of the labeling data is input into the machine learning model, and the parameters of the machine learning model itself are adjusted until the position output by the machine learning model is the position in the labeling data or converges to the position in the labeling data, and the training is completed to obtain the dynamic detection model of the hand key points.

[0088] It can be seen that in at least one data labeling device embodiment of the present invention, by using multiple models to detect the positions of hand key points in the image, the hand key point sets corresponding to the multiple models can be obtained; then by calculating the relative position deviation between these hand key point sets, if the relative position deviation is less than or equal to the preset relative position deviation threshold, it means that the position of the hand key point detected by the multiple models is close to the true position, then the position information of each hand key point is obtained by comprehensively considering these hand key point sets, which represents the true position of each hand key point. Therefore, using the position information of each hand key point as the labeling data of the image can reduce the labeling error, thereby reducing the jitter of the hand key points and improving the stability of the hand key points.

[0089] Figure 3 is an exemplary block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 3 As shown, the electronic device includes: a memory 31, a processor 32, and a computer program stored in the memory 31. It can be understood that the memory 31 in this embodiment can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories.

[0090] In some implementations, the memory 31 stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system and application programs.

[0091] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic tasks and process hardware-based tasks. The application includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application tasks. The program that implements the data annotation method provided by the embodiment of the present disclosure can be included in the application.

[0092] In the embodiment of the present disclosure, at least one processor 32 calls a program or instruction stored in at least one memory 31, specifically, a program or instruction stored in an application program, and at least one processor 32 is used to execute the steps of each embodiment of the data labeling method provided in the embodiment of the present disclosure, for example, executing the following steps:

[0093] Get the image containing the gesture;

[0094] Detect hand key points of the image based on at least two hand key point detection models respectively, and obtain a hand key point set corresponding to each detection model;

[0095] Calculate the relative position deviation between the hand key point sets corresponding to each detection model;

[0096] If the relative position deviation between the hand key point sets corresponding to each detection model is less than or equal to a preset relative position deviation threshold, the position information of the hand key points in the image is determined based on the hand key point sets corresponding to each detection model as the image annotation data.

[0097] To avoid repeated description, it will not be repeated here.

[0098] The data annotation method provided in the embodiment of the present disclosure can be applied to the processor 32, or implemented by the processor 32. The processor 32 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 32 or an instruction in the form of software. The above-mentioned processor 32 can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0099] The steps of the data annotation method provided in the embodiment of the present disclosure can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 31, and the processor 32 reads the information in the memory 31 and completes the steps of the method in combination with its hardware.

[0100] The present disclosure also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program or instruction, wherein the program or instruction enables a computer to execute the steps of each embodiment of the data labeling method, for example, to execute the following steps:

[0101] Get the image containing the gesture;

[0102] Detect hand key points of the image based on at least two hand key point detection models respectively, and obtain a hand key point set corresponding to each detection model;

[0103] Calculate the relative position deviation between the hand key point sets corresponding to each detection model;

[0104] If the relative position deviation between the hand key point sets corresponding to each detection model is less than or equal to a preset relative position deviation threshold, the position information of the hand key points in the image is determined based on the hand key point sets corresponding to each detection model as the image annotation data.

[0105] To avoid repeated description, it will not be described here. The computer-readable storage medium may be a non-transitory computer-readable storage medium.

[0106] The present disclosure also provides a computer program product, which includes a computer program stored in a computer-readable storage medium, which may be a non-transitory computer-readable storage medium. At least one processor of a computer reads and executes the computer program from the computer-readable storage medium, so that the computer performs the steps of each embodiment of the data labeling method, for example, the following steps:

[0107] Get the image containing the gesture;

[0108] Detect hand key points of the image based on at least two hand key point detection models respectively, and obtain a hand key point set corresponding to each detection model;

[0109] Calculate the relative position deviation between the hand key point sets corresponding to each detection model;

[0110] If the relative position deviation between the hand key point sets corresponding to each detection model is less than or equal to a preset relative position deviation threshold, the position information of the hand key points in the image is determined based on the hand key point sets corresponding to each detection model as the image annotation data.

[0111] To avoid repeated description, it will not be repeated here.

[0112] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0113] Those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present disclosure and to form different embodiments.

[0114] Those skilled in the art will appreciate that the description of each embodiment has its own emphasis, and for parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0115] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A data annotation method, the method comprising: Get the image containing the gesture; Based on at least two detection models of hand key points, respectively detect hand key points of the image to obtain a set of hand key points corresponding to each detection model; Calculating the relative position deviation between the hand key point sets corresponding to each of the detection models; If the relative position deviation between the hand key point sets corresponding to each of the detection models is less than or equal to a preset relative position deviation threshold, the position information of the hand key points in the image is determined based on the hand key point sets corresponding to each of the detection models as the annotation data of the image.

2. The method according to claim 1, wherein: The at least two hand key point detection models respectively adopt one or more different trained models in model architecture, image resolution, input image size and backbone network.

3. The method according to claim 1, wherein: Before calculating the relative position deviation between the hand key point sets corresponding to each of the detection models, the method further includes: The hand key point set corresponding to each detection model is subjected to background false detection filtering to filter out the points located on the image background as hand key points.

4. The method according to claim 1, wherein: The calculating the relative position deviation between the hand key point sets corresponding to each of the detection models comprises: Based on the position of each hand key point in the hand key point set corresponding to each detection model, calculating the Euclidean distance between the position of each hand key point in different hand key point sets; The average value of the Euclidean distances of the positions of the hand key points in different sets of hand key points is calculated to obtain the average Euclidean distance as the relative position deviation.

5. The method according to claim 4, wherein: If the relative position deviation between the hand key point sets corresponding to each of the detection models is less than or equal to a preset relative position deviation threshold, determining the position information of the hand key points in the image as the annotation data of the image based on the hand key point sets corresponding to each of the detection models, comprising: If the average Euclidean distance is less than or equal to a preset average Euclidean distance threshold, the average value of the positions of the hand key points in different hand key point sets is used as the position information of the hand key points, and the position information of the hand key points is used as the annotation data of the image.

6. The method according to claim 4, wherein: After obtaining the average Euclidean distance, the method further includes: If the average Euclidean distance is greater than a preset average Euclidean distance threshold, the hand key point set corresponding to each detection model is associated and stored in a file to be manually reviewed.

7. The method according to claim 1, wherein: After determining the position information of the hand key points in the image as the annotation data of the image based on the hand key point set corresponding to each of the detection models, the method further includes: The image where the key points of the hand as the annotation data are located is used as the associated image of the annotation data; When training a dynamic detection model for hand key points, the associated image of the annotated data is input into the machine learning model, and the parameters of the machine learning model itself are adjusted until the position output by the machine learning model is the position in the annotated data or converges to the position in the annotated data. The training is completed to obtain a dynamic detection model for hand key points.

8. A data annotation device, comprising: An acquisition unit, used for acquiring an image containing a gesture; A detection unit, configured to detect hand key points of the image based on at least two detection models of hand key points, and obtain a set of hand key points corresponding to each detection model; A calculation unit, used for calculating the relative position deviation between the hand key point sets corresponding to each of the detection models; A determination unit is used to determine the position information of the hand key points in the image as the annotation data of the image based on the hand key point sets corresponding to each of the detection models if the relative position deviation between the hand key point sets corresponding to each of the detection models is less than or equal to a preset relative position deviation threshold.

9. An electronic device, wherein: The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the data labeling method according to any one of claims 1 to 7.

10. A computer-readable storage medium, wherein: The computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute the steps of the data labeling method according to any one of claims 1 to 7.

Images