Data annotation method and device, electronic equipment and storage medium

Abstract

The invention discloses a data annotation method and device, electronic equipment and a storage medium. The method comprises the steps that multiple pieces of to-be-annotated data are acquired; reasoning each piece of to-be-labeled data by utilizing a labeling model, and outputting a reasoning result corresponding to each piece of to-be-labeled data; and according to the reasoning results corresponding to the to-be-labeled data, performing data labeling on the to-be-labeled data. According to the method, effective data annotation can be carried out, manual annotation work is reduced, the data annotation efficiency is improved, and the method can be widely applied to the technical field of artificial intelligence.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data labeling method, device, electronic device and storage medium. Background Art

[0002] Data labeling is an important task in the field of artificial intelligence. Data labeling means adding reliable label information to data. For example, in image classification tasks, data labeling can refer to marking the true category of the training image, so that the true category can be used as a supervisory signal to conduct supervised training on the image classification model, so that the model can learn the correspondence between the content of the training image and its true category, and then effectively classify other images.

[0003] However, current data labeling mainly relies on a large amount of manpower input, which leads to very low efficiency of data labeling and consumes a lot of time and energy. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a data labeling method, device, electronic device and storage medium, which can perform effective data labeling and improve the efficiency of data labeling.

[0005] On the one hand, an embodiment of the present application proposes a data labeling method, the method comprising the following steps:

[0006] Obtain multiple data to be labeled;

[0007] Using the annotation model to infer the data to be annotated, and outputting the inference results corresponding to the data to be annotated;

[0008] Data labeling is performed on each of the data to be labeled according to the inference results corresponding to each of the data to be labeled.

[0009] In some embodiments, the using of the annotation model to infer each of the data to be annotated and outputting the inference result corresponding to each of the data to be annotated specifically includes:

[0010] Each of the data to be labeled is input into the labeling model, the labeling model is used to perform data prediction on each of the data to be labeled, and the inference label and inference confidence corresponding to each of the data to be labeled are output.

[0011] In some embodiments, the step of labeling each of the data to be labeled according to the inference result corresponding to each of the data to be labeled specifically includes:

[0012] Determining whether to perform data labeling on the model to be labeled according to the inference label based on the inference confidence corresponding to the data to be labeled;

[0013] When the reasoning confidence is less than a given confidence threshold, the data to be labeled is determined as labeling uncertain data and stored in a third labeled data set;

[0014] When the inference confidence is greater than the confidence threshold, the data to be labeled is labeled using the inference label to obtain corresponding second labeled data, and the second labeled data is stored in a second labeled data set.

[0015] In some embodiments, the method further comprises:

[0016] Acquire a first annotated data set; the first annotated data set includes a plurality of manually annotated first annotated data;

[0017] The annotation model is trained using the first annotation data set and the second annotation data set to determine a plurality of annotation difference data and an inference difference result corresponding to each of the annotation difference data, and each of the annotation difference data and the corresponding inference difference result is stored in the third annotation data set, wherein the inference difference result includes a previous training label of the annotation difference data, a current training label, and a current inference confidence corresponding to the current training label.

[0018] In some embodiments, the using the first annotated data set and the second annotated data set to train the annotation model, determining a plurality of annotated difference data and reasoning difference results corresponding to each of the annotated difference data, and storing each of the annotated difference data and the corresponding reasoning difference results in the third annotated data set specifically includes:

[0019] Dynamically obtain a current data change ratio, where the current data change ratio is a ratio of the amount of data with data changes in the first annotated data set and the second annotated data set to the amount of data for last model training, where the amount of data for last model training is the total amount of training data in the first annotated data set and the second annotated data set used for training the annotated model during the last annotated model training;

[0020] When the current data change ratio exceeds a preset ratio threshold, a plurality of training data are determined from the first annotated data set and the second annotated data set, a training label corresponding to each of the training data is obtained as the previous training label, each of the training data is input into the annotation model, and the annotation model is used to output the current training label and the current reasoning confidence corresponding to each of the training data;

[0021] When the current training label corresponding to the training data is inconsistent with the previous training label, the training data is determined as the labeled difference data, and the labeled difference data is bound to the previous training label, the current training label and the current reasoning confidence, and then stored in the third labeled data set.

[0022] In some embodiments, the method further comprises:

[0023] According to a preset data selection strategy, determining a plurality of inference difference data from the third annotated data set;

[0024] In response to the manual labeling operation on each of the reasoning difference data, the optimal reasoning label of each of the reasoning difference data is determined, the reasoning difference data is labeled using the optimal reasoning label, the labeled determination data corresponding to each of the reasoning difference data is obtained, and each of the labeled determination data is stored in the first labeled data set.

[0025] In some embodiments, determining a plurality of inference difference data from the third annotated data set according to a preset data selection strategy specifically includes:

[0026] Filtering out a plurality of the labeled difference data and a plurality of the labeled uncertain data from the third labeled data set;

[0027] According to the reasoning confidence corresponding to each of the labeled uncertain data, the data are sorted in ascending order, and a plurality of first reasoning difference data are determined from the plurality of labeled uncertain data, wherein the first reasoning difference data is the labeled uncertain data whose ranking in ascending order is less than a preset first ranking threshold;

[0028] According to the current reasoning confidence of each of the labeled difference data, descending sorting is performed, and a plurality of second reasoning difference data are determined from the plurality of labeled difference data, wherein the second reasoning difference data are the labeled difference data whose ranking in descending sorting is less than a preset second ranking threshold.

[0029] On the other hand, an embodiment of the present application provides a data labeling device, the device comprising:

[0030] The first module is used to obtain multiple data to be labeled;

[0031] The second module is used to use the annotation model to infer the data to be annotated, and output the inference results corresponding to the data to be annotated;

[0032] The third module is used to label each of the data to be labeled according to the inference results corresponding to each of the data to be labeled.

[0033] On the other hand, an embodiment of the present application provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the data labeling method described above when executing the computer program.

[0034] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the data labeling method described above is implemented.

[0035] The embodiments of the present application include at least the following beneficial effects: the present application provides a data annotation method, device, electronic device and storage medium, which obtains multiple data to be annotated; uses the annotation model to infer each data to be annotated, outputs the inference result corresponding to each data to be annotated; and annotates each data to be annotated according to the inference result corresponding to each data to be annotated. The present application can perform effective data annotation, reduce manual annotation work, and improve the efficiency of data annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0038] Figure 1 is a flow chart of a data labeling method provided in an embodiment of the present application;

[0039] Figure 2 This is a schematic diagram of the structure of semi-automatic annotation in an embodiment of the present application;

[0040] Figure 3 It is a structural schematic diagram of a data labeling device provided in an embodiment of the present application;

[0041] Figure 4 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0043] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0044] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0046] At present, manual data labeling consumes a lot of manpower in both data labeling and model training. Data engineers need to manually check each image and label it manually. Manual labeling includes directly using folder classification or using labeling tools to label. The workload of manual labeling is huge. When training models, AI engineers need to constantly adjust data and repeat training according to actual conditions to obtain a better model. This process of continuous training, reasoning, and adjusting data consumes a lot of time and energy.

[0047] Based on this, the embodiments of the present application propose a data labeling method, device, electronic device and storage medium, which can achieve data labeling through semi-automatic labeling, reduce excessive participation of AI engineers, and improve the efficiency of data labeling.

[0048] Reference Figure 1, Figure 1 This is an optional flowchart of a data annotation method provided in an embodiment of the present application. The method may include but is not limited to steps S101 to S103:

[0049] Step S101, obtaining a plurality of data to be labeled;

[0050] Step S102, using the annotation model to infer each data to be annotated, and outputting the inference result corresponding to each data to be annotated;

[0051] Step S103: labeling each data to be labeled according to the inference result corresponding to each data to be labeled.

[0052] In some embodiments, the first labeled data set is used to store multiple first labeled data that have been manually labeled and have high credibility, the second labeled data set is used to store second labeled data that have undergone model reasoning and obtained corresponding reasoning labels and have high reasoning confidence, the third labeled data set is used to store data waiting for manual labeling, including labeled difference data and reasoning difference data, and the fourth labeled data set is used to store multiple data that have not been manually labeled or model reasoning, that is, the above-mentioned data to be labeled.

[0053] In step S102 of some embodiments, the inference result includes an inference label and an inference confidence. Specifically, each data to be labeled is input into the labeling model, and the labeling model is used to perform data prediction on each data to be labeled, and the inference label and inference confidence corresponding to each data to be labeled are output.

[0054] In step S103 of some embodiments, based on the inference confidence corresponding to the data to be labeled, it is determined whether to perform data labeling on the model to be labeled based on the inference label. Optionally, when the inference confidence is less than a given confidence threshold, the data to be labeled is determined as labeled uncertain data and stored in a third labeled data set; when the inference confidence is greater than the confidence threshold, the data to be labeled is labeled using the inference label to obtain corresponding second labeled data, and the second labeled data is stored in the second labeled data set.

[0055] When the inference confidence of the data to be labeled is less than a given confidence threshold, it means that the model cannot accurately determine the inference label of the data to be labeled. Therefore, the data to be labeled is determined as uncertain labeling data and then stored in the third labeled data set, waiting for manual labeling.

[0056] In some embodiments, the above data annotation method may also include but is not limited to steps S201 to S202:

[0057] Step S201, obtaining a first annotated data set; the first annotated data set includes a plurality of manually annotated first annotated data;

[0058] Step S202, use the first labeled data set and the second labeled data set to train the labeling model, determine multiple labeled difference data and the reasoning difference results corresponding to each labeled difference data, store each labeled difference data and the corresponding reasoning difference result in a third labeled data set, and the reasoning difference result includes the previous training label of the labeled difference data, the current training label and the current reasoning confidence corresponding to the current training label.

[0059] In some embodiments, the annotation model is automatically trained using the first annotation data set and the second annotation data set to determine multiple annotation difference data. Optionally, the training data is the first annotation data or the second annotation data, and the training label of the training data at this time is the previous training label. The training data is then input into the annotation model, and the annotation model is used to perform model inference. The inference label and inference confidence of the training data are output, and the inference label is determined as the current training label of the training data, and the inference confidence is determined as the current inference confidence of the training data.

[0060] In some embodiments, step S202 may include but is not limited to steps S301 to S303:

[0061] Step S301, dynamically obtain the current data change ratio, the current data change ratio is the ratio of the amount of data with data changes in the first labeled data set and the second labeled data set to the amount of data for the previous model training, the amount of data for the previous model training is the total amount of training data in the first labeled data set and the second labeled data set used for training the labeled model during the last labeled model training;

[0062] Step S302, when the current data change ratio exceeds a preset ratio threshold, multiple training data are determined from the first labeled data set and the second labeled data set, a training label corresponding to each training data is obtained as a previous training label, each training data is input into a labeling model, and the labeling model is used to output a current training label and a current reasoning confidence corresponding to each training data;

[0063] Step S303, when the current training label corresponding to the training data is inconsistent with the previous training label, the training data is determined as labeled difference data, the labeled difference data is bound to the previous training label, the current training label and the current reasoning confidence, and then stored in a third labeled data set.

[0064] In some embodiments, data changes may include, but are not limited to, data modification and data addition. For example, when data addition occurs in the first labeled data set or the second labeled data set, such as multiple first labeled data are added to the first labeled data set, the amount of newly added data M1 is recorded. When data modification occurs in the first labeled data set or the second labeled data set, such as a change in the inference label of the data, the amount of data M2 of the modified information is recorded, and the amount of data M2 of the previous model training data is obtained. c and ratio threshold M G , when (M1+M2) / M c >M G When the first labeled data set and the second labeled data set are used, a plurality of training data are randomly selected again, and a new round of model training is performed on the labeled model using the plurality of training data.

[0065] In some embodiments, when the current training label corresponding to the training data is inconsistent with the previous training label, it means that there is a difference between the manual annotation result of the training data and the model inference result, or the model inference results before and after are inconsistent. Therefore, the training data needs to be manually annotated again, and the training data is determined as the annotated difference data, and then stored in the third annotated data set.

[0066] In some embodiments, the above data annotation method may also include but is not limited to steps S401 to S402:

[0067] Step S401, determining a plurality of inference difference data from a third annotated data set according to a preset data selection strategy;

[0068] Step S402, in response to the manual labeling operation on each inference difference data, determine the optimal inference label for each inference difference data, use the optimal inference label to label the inference difference data, obtain the labeling determination data corresponding to each inference difference data, and store each labeling determination data in the first labeling data set.

[0069] In some embodiments, according to the data selection strategy, multiple inference difference data with large differences are selected from the third labeled data set. The inference difference data is labeled difference data or labeled uncertain data. Manual labeling is used to determine the optimal inference label for the inference difference data. The optimal inference label is used to label the inference difference data to obtain corresponding labeled determined data. The labeled determined data can more effectively train the model and improve the accuracy of model reasoning.

[0070] In some embodiments, reference Figure 2 , Figure 2It is an optional structural diagram of semi-automatic labeling in an embodiment of the present application, wherein a plurality of labeling difference data are determined from a first labeling data set and a second data set, a plurality of labeling uncertain data are determined from a fourth labeling data set, each standard difference data and each labeling uncertain data are stored in a third labeling data set, and then a plurality of inference difference data are determined from the third labeling data set, each inference difference data is manually labeled by manual labeling to obtain corresponding each labeling determination data, and then each labeling determination data is stored in the first labeling data set again.

[0071] In some embodiments, step S401 may include but is not limited to steps S501 to S503:

[0072] Step S501, screening out a plurality of labeled difference data and a plurality of labeled uncertain data from a third labeled data set;

[0073] Step S502, performing ascending sorting according to the inference confidence corresponding to each labeled uncertain data, and determining a plurality of first inference difference data from the plurality of labeled uncertain data, wherein the first inference difference data is the labeled uncertain data whose ranking in ascending sorting is less than a preset first ranking threshold;

[0074] Step S503 , sorting in descending order according to the current reasoning confidence of each labeled difference data, and determining a plurality of second reasoning difference data from the plurality of labeled difference data, wherein the second reasoning difference data is labeled difference data whose ranking in descending order is less than a preset second ranking threshold.

[0075] In some embodiments, the inference confidence corresponding to each labeled uncertain data is sorted in ascending order, that is, the inference confidence corresponding to each labeled uncertain data is sorted from low to high, and according to a preset first ranking threshold, all labeled uncertain data before the first ranking threshold are taken as inference difference data. In this way, the labeled uncertain data with lower inference confidence can be effectively screened out.

[0076] In some embodiments, each labeled difference data is sorted in descending order according to its current reasoning confidence, that is, it is sorted from high to low according to its current reasoning confidence, and according to a preset second ranking threshold, all labeled difference data before the second ranking threshold are taken as reasoning difference data. The higher the current reasoning confidence of the labeled difference data, the greater the difference between the previous training label and the current training label of the labeled difference data. In this way, the labeled difference data with large difference in training labels can be effectively screened out.

[0077] In some embodiments, if the third annotated data set includes multiple image data, the data selection strategy may also include an image similarity screening strategy, which calculates the image similarity between each image data and other image data, and screens out a number of target image data from the multiple image data. The image similarities between the target image data and the other image data are all lower than the similarity threshold. The target image data is determined as inference difference data, and data is annotated on the target image data using manual annotation.

[0078] Reference Figure 3 , Figure 3 is an optional structural diagram of a data labeling device provided in an embodiment of the present application. The device is used to implement the above-mentioned data labeling method. The device may include:

[0079] The first module is used to obtain multiple data to be labeled;

[0080] The second module is used to use the annotation model to infer each data to be annotated, and output the inference result corresponding to each data to be annotated;

[0081] The third module is used to label each data to be labeled according to the inference results corresponding to each data to be labeled.

[0082] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0083] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above data labeling method when executing the computer program. The electronic device can be any smart terminal including a tablet computer.

[0084] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0085] See also Figure 4 , Figure 4 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0086] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0087] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the data labeling method of the embodiment of this application;

[0088] Input / output interface 903, used to implement information input and output;

[0089] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0090] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0091] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0092] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned data labeling method is implemented.

[0093] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0094] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0095] The embodiments of the present application provide a data annotation method, device, electronic device, and storage medium, which obtain multiple data to be annotated; use an annotation model to infer each data to be annotated, output the inference result corresponding to each data to be annotated; and perform data annotation on each data to be annotated according to the inference result corresponding to each data to be annotated. The embodiments of the present application can perform effective data annotation, reduce manual annotation work, and improve the efficiency of data annotation.

[0096] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0097] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0098] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0099] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0100] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0101] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0102] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0103] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0104] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0105] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method may be implemented in a computer program using standard programming techniques including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in an assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed ASIC for this purpose.

[0106] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0107] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A data labeling method, characterized in that: The method comprises the following steps: Obtain multiple data to be labeled; Using the annotation model to infer the data to be annotated, and outputting the inference results corresponding to the data to be annotated; Data labeling is performed on each of the data to be labeled according to the inference results corresponding to each of the data to be labeled.

2. The data labeling method according to claim 1, characterized in that: The using the annotation model to reason on each of the to-be-annotated data and outputting the reasoning result corresponding to each of the to-be-annotated data specifically includes: Each of the data to be labeled is input into the labeling model, the labeling model is used to perform data prediction on each of the data to be labeled, and the inference label and inference confidence corresponding to each of the data to be labeled are output.

3. The data labeling method according to claim 2, characterized in that: The step of labeling each of the data to be labeled according to the inference result corresponding to each of the data to be labeled specifically includes: Determining whether to perform data labeling on the model to be labeled according to the inference label based on the inference confidence corresponding to the data to be labeled; When the reasoning confidence is less than a given confidence threshold, the data to be labeled is determined as labeling uncertain data and stored in a third labeled data set; When the inference confidence is greater than the confidence threshold, the data to be labeled is labeled using the inference label to obtain corresponding second labeled data, and the second labeled data is stored in a second labeled data set.

4. The data labeling method according to claim 3, characterized in that: The method further comprises: Acquire a first annotated data set; the first annotated data set includes a plurality of manually annotated first annotated data; The annotation model is trained using the first annotation data set and the second annotation data set to determine a plurality of annotation difference data and an inference difference result corresponding to each of the annotation difference data, and each of the annotation difference data and the corresponding inference difference result is stored in the third annotation data set, wherein the inference difference result includes a previous training label of the annotation difference data, a current training label, and a current inference confidence corresponding to the current training label.

5. The data labeling method according to claim 4, characterized in that: The method of training the annotation model by using the first annotation data set and the second annotation data set, determining a plurality of annotation difference data and reasoning difference results corresponding to each of the annotation difference data, and storing each of the annotation difference data and the corresponding reasoning difference results in the third annotation data set specifically includes: Dynamically obtain a current data change ratio, where the current data change ratio is a ratio of the amount of data with data changes in the first annotated data set and the second annotated data set to the amount of data for last model training, where the amount of data for last model training is the total amount of training data in the first annotated data set and the second annotated data set used for training the annotated model during the last annotated model training; When the current data change ratio exceeds a preset ratio threshold, a plurality of training data are determined from the first annotated data set and the second annotated data set, a training label corresponding to each of the training data is obtained as the previous training label, each of the training data is input into the annotation model, and the annotation model is used to output the current training label and the current reasoning confidence corresponding to each of the training data; When the current training label corresponding to the training data is inconsistent with the previous training label, the training data is determined as the labeled difference data, and the labeled difference data is bound to the previous training label, the current training label and the current reasoning confidence, and then stored in the third labeled data set.

6. The data labeling method according to claim 4, characterized in that: The method further comprises: According to a preset data selection strategy, determining a plurality of inference difference data from the third annotated data set; In response to the manual labeling operation on each of the reasoning difference data, the optimal reasoning label of each of the reasoning difference data is determined, the reasoning difference data is labeled using the optimal reasoning label, the labeled determination data corresponding to each of the reasoning difference data is obtained, and each of the labeled determination data is stored in the first labeled data set.

7. The data labeling method according to claim 6, characterized in that: The step of determining a plurality of inference difference data from the third annotated data set according to a preset data selection strategy specifically includes: Filtering out a plurality of the labeled difference data and a plurality of the labeled uncertain data from the third labeled data set; According to the reasoning confidence corresponding to each of the labeled uncertain data, the data are sorted in ascending order, and a plurality of first reasoning difference data are determined from the plurality of labeled uncertain data, wherein the first reasoning difference data is the labeled uncertain data whose ranking in ascending order is less than a preset first ranking threshold; According to the current reasoning confidence of each of the labeled difference data, descending sorting is performed, and a plurality of second reasoning difference data are determined from the plurality of labeled difference data, wherein the second reasoning difference data are the labeled difference data whose ranking in descending sorting is less than a preset second ranking threshold.

8. A data labeling device, characterized in that: The device comprises: The first module is used to obtain multiple data to be labeled; The second module is used to use the annotation model to infer the data to be annotated, and output the inference results corresponding to the data to be annotated; The third module is used to label each of the data to be labeled according to the inference results corresponding to each of the data to be labeled.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the data labeling method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data labeling method according to any one of claims 1 to 7 is implemented.

Images