Data annotation method and device, electronic equipment and storage medium

Abstract

The invention provides a data annotation method and device, electronic equipment and a storage medium. The method comprises the steps that a data annotation request is received, the data annotation request comprises at least one annotation task and a task type, annotation data, a constraint and an annotation index value corresponding to each annotation task, the annotation data comprises at least one data item, and the constraint comprises at least one constraint condition; determining a target labeling task according to the labeling index value of each labeling task; dividing annotation data of the target annotation task into automatic annotation data and manual annotation data according to the constraint; determining a target automatic labeling strategy according to at least one of the task type, the labeling quality and the saved cost of the to-be-labeled task; and labeling the automatic labeling data through the target automatic labeling strategy. By dynamically balancing multiple target values such as data annotation quality, annotation efficiency and labor cost, better resource allocation and personalized data annotation configuration for each annotation task are realized.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a data labeling method, device, electronic device, and storage medium. Background Art

[0002] Data annotation refers to the process of providing annotations for data such as speech, images, text, and video, and converting them into information that can be recognized by machine learning algorithms. Using data annotation can enable machine learning algorithms to establish an accurate understanding of real-world environments and conditions.

[0003] In the prior art, methods for implementing data labeling can be divided into manual labeling and automatic labeling. Manual labeling refers to manual labeling of data by labelers, and automatic labeling refers to labeling of data by machine learning models through predicting data labels. It is understandable that although manual labeling has higher data labeling quality, it will also generate a large amount of labeling fees and labor costs. Although the labeling quality of automatic labeling is lower than that of manual labeling, it can reduce labeling fees and labor costs. In addition, due to issues such as data security, the labeling requirements for different labeling tasks are different, and the data labeling methods that need to be selected are also different. Therefore, how to choose the data labeling method and how to balance multiple goals such as data labeling quality, labeling efficiency, and labor costs are crucial.

[0004] Therefore, it is necessary to propose a new data annotation method to solve at least one of the above technical problems. Summary of the invention

[0005] The embodiments of the present disclosure provide a data labeling method, device, electronic device, and storage medium.

[0006] In a first aspect, the present disclosure provides a data annotation method, comprising:

[0007] Receive a data annotation request, the data annotation request including at least one annotation task and a task type, annotation data, constraints, and annotation index values ​​corresponding to each annotation task, the annotation data including at least one data item, and the constraints including at least one constraint condition;

[0008] Determine the target labeling task according to the labeling indicator value of each labeling task;

[0009] According to the above constraints, the above-mentioned labeled data of the above-mentioned target labeling task is divided into automatically labeled data and manually labeled data;

[0010] Determine a target automatic labeling strategy according to at least one of a task type, labeling quality, and cost saving of the task to be labeled;

[0011] The above automatic labeling data is labeled using the above target automatic labeling strategy.

[0012] In some optional implementations, the above-mentioned determining the target automatic labeling strategy according to at least one of the task type, labeling quality and cost saving of the task to be labeled includes:

[0013] Determine, according to the task type of the target labeling task, whether there is a historical automatic labeling strategy corresponding to the target labeling task;

[0014] If so, determining the above historical automatic labeling strategy as the target historical automatic labeling strategy, and determining whether the above target historical automatic labeling strategy meets the first quality assessment indicator;

[0015] If not, obtaining an automatic labeling strategy set according to the task type of the target labeling task, wherein the automatic labeling strategy set includes at least one automatic labeling strategy;

[0016] Based on at least one automatic labeling strategy, a target automatic labeling strategy is determined.

[0017] In some optional implementations, the determining of the target automatic labeling strategy based on at least one automatic labeling strategy includes:

[0018] Calculate the annotation quality and cost savings of each automatic annotation strategy;

[0019] Determine a first automatic labeling strategy according to the labeling quality and the cost saving;

[0020] Determining whether the first automatic labeling strategy meets the first quality assessment indicator;

[0021] If satisfied, the first automatic labeling strategy is determined as the target automatic labeling strategy.

[0022] In some optional implementations, determining the target labeling task according to the labeling indicator value of each labeling task includes:

[0023] Calculating a first number of remaining labeling tasks dominated by each labeling task according to the labeling indicator value of each labeling task, where the remaining labeling tasks refer to the labeling tasks remaining after selecting any labeling task from the at least one labeling task;

[0024] Priority is sorted for the at least one labeling task according to the first number, and the labeling task with the highest first number is determined as the target labeling task.

[0025] In some optional implementations, dividing the labeled data of the target labeling task into automatically labeled data and manually labeled data according to the constraints includes:

[0026] Determine whether each data item in the labeled data satisfies each constraint condition of the constraint corresponding to the target labeling task;

[0027] If satisfied, the data item of the annotated data is determined as automatically annotated data;

[0028] If not satisfied, the data item of the above-mentioned labeled data is determined as manually labeled data.

[0029] In some optional implementations, the step of determining the data item of the annotated data as manually annotated data further includes:

[0030] Determine the target pre-filtering strategy according to the task type of the above target labeling task;

[0031] According to the above target pre-filtering strategy, each data item in the above manually labeled data is judged to determine whether it meets the target pre-filtering strategy threshold;

[0032] If satisfied, the above data item is sent to the manually labeled data item queue;

[0033] If not satisfied, the above data item is determined as a negative example and discarded.

[0034] In some optional implementations, the target pre-filtering strategy is determined according to the task type of the target labeling task, including:

[0035] Obtain a pre-filtering strategy set, the pre-filtering strategy set including at least one pre-filtering strategy, each of the pre-filtering strategies corresponding to a task type of a filterable labeling task, and the pre-filtering strategy is used to discard negative examples in the manually labeled data;

[0036] Matching the task type of the target labeling task with the task type of the filterable labeling task corresponding to the pre-filtering strategy, and determining whether the task type of the target labeling task exists in the task type of the filterable labeling task corresponding to the pre-filtering strategy;

[0037] If so, the pre-filtering strategy corresponding to the task type of the target labeling task is determined as the target pre-filtering strategy.

[0038] In some optional implementations, determining whether there is a historical automatic labeling strategy corresponding to the target labeling task according to the task type of the target labeling task includes:

[0039] Obtain a historical automatic annotation strategy set, the historical automatic annotation strategy set includes at least one historical automatic annotation strategy, each of the historical automatic annotation strategies corresponds to a task type of an annotable task; the historical automatic annotation strategy is used to annotate the automatically annotated data;

[0040] Matching the task type of the target labeling task with the task type of the labelable task corresponding to the historical automatic labeling strategy;

[0041] It is determined whether there is the historical automatic labeling strategy corresponding to the target labeling task according to whether there is a task type of the target labeling task among the task types of labelable tasks corresponding to the historical automatic labeling strategy.

[0042] In some optional implementations, determining whether there is the above-mentioned historical automatic labeling strategy corresponding to the above-mentioned target labeling task according to whether there is the task type of the above-mentioned target labeling task among the task types of the labelable tasks corresponding to the above-mentioned historical automatic labeling strategy, further includes:

[0043] If it does not exist, the above automatic annotation data will be determined as manual annotation data.

[0044] In some optional implementations, the above-mentioned determining whether the above-mentioned target history automatic labeling strategy meets the first quality assessment indicator includes:

[0045] Randomly extract data with a preset quality inspection ratio from the automatically labeled data, and determine the data with the preset quality inspection ratio as quality inspection data;

[0046] Use the above-mentioned target history automatic labeling strategy to label the above-mentioned quality inspection data;

[0047] Obtain the labeling results of the above-mentioned target history automatic labeling strategy on the above-mentioned quality inspection data and the manual labeling results of the above-mentioned quality inspection data;

[0048] According to the labeling result of the quality inspection data by the target history automatic labeling strategy and the manual labeling result of the quality inspection data, it is judged whether the target history automatic labeling strategy meets the first quality assessment indicator.

[0049] In some optional implementations, the first quality assessment indicator includes recall rate, precision rate, accuracy rate, false positive rate and false negative rate; and

[0050] The above-mentioned judging whether the target history automatic labeling strategy meets the first quality assessment indicator according to the labeling result of the quality inspection data by the target history automatic labeling strategy and the manual labeling result of the quality inspection data includes:

[0051] Calculate the recall rate, the precision rate, the accuracy rate, the false positive rate, and the false negative rate based on the labeling results of the quality inspection data by the target history automatic labeling strategy and the manual labeling results of the quality inspection data;

[0052] Determine whether the target history automatic labeling strategy meets the first quality assessment indicator according to whether the recall rate, the precision rate, the accuracy rate, the false positive rate, and the false negative rate are respectively greater than the recall rate threshold, the precision rate threshold, the accuracy rate threshold, the false positive rate threshold, and the false negative rate threshold;

[0053] If the recall rate, the precision rate, the accuracy rate, the false positive rate, and the false negative rate are respectively greater than the recall rate threshold, the precision rate threshold, the accuracy rate threshold, the false positive rate threshold, and the false negative rate threshold, it is determined that the target history automatic labeling strategy meets the first quality evaluation indicator;

[0054] If at least one of the recall rate, the precision rate, the accuracy rate, the false positive rate and the false negative rate is not greater than the recall rate threshold, the precision rate threshold, the accuracy rate threshold, the false positive rate threshold and the false negative rate threshold, it is determined that the target history automatic labeling strategy does not meet the first quality evaluation indicator.

[0055] In some optional implementations, the above-mentioned determination of whether the above-mentioned target history automatic labeling strategy meets the first quality assessment indicator further includes:

[0056] If satisfied, the above automatically labeled data is labeled using the above target history automatic labeling strategy.

[0057] In some optional implementations, the step of obtaining the automatic labeling strategy set according to the task type of the target labeling task includes:

[0058] According to the task type of the above target labeling task, select X feature extraction models and Y classifier models from the predefined feature extraction model library and classifier model library respectively, where X and Y are both positive integers;

[0059] Any feature extraction model among the X feature extraction models and any classifier model among the Y classifier models are arranged and combined to obtain an automatic labeling strategy set.

[0060] In some optional implementations, the above calculation of the annotation quality and cost saving of each automatic annotation strategy includes:

[0061] Using each of the above automatic labeling strategies to label the above quality inspection data respectively;

[0062] Calculate the annotation quality of each of the automatic annotation strategies according to the annotation results of the quality inspection data by each of the automatic annotation strategies and the manual annotation results of the quality inspection data;

[0063] Obtaining task characteristics of the target labeling task, the task characteristics including the number of data items of the automatically labeled data and the average cost of manually labeling a single data item of the automatically labeled data;

[0064] The cost saving of each automatic labeling strategy is calculated according to the task characteristics and the labeling results of the quality inspection data by each automatic labeling strategy.

[0065] In some optional implementations, the above-mentioned calculating the annotation quality of each of the above-mentioned automatic annotation strategies according to the annotation results of the above-mentioned quality inspection data by each of the above-mentioned automatic annotation strategies and the manual annotation results of the above-mentioned quality inspection data includes:

[0066] The recall rate, the precision rate, the accuracy rate and the proportion of correctly identified negative examples of each automatic labeling strategy are calculated respectively according to the labeling results of the quality inspection data by each automatic labeling strategy and the manual labeling results of the quality inspection data.

[0067] The recall rate, the precision rate, the accuracy rate and the ratio of correctly identified negative examples are linearly weighted to obtain the labeling quality of each of the automatic labeling strategies.

[0068] In some optional implementations, the determining of the first automatic labeling strategy according to the labeling quality and the cost saving includes:

[0069] Based on the above annotation quality and the above cost saving, calculate the Pareto dominance relationship between any two of the above automatic annotation strategies;

[0070] Based on the above Pareto dominance relationship, the first automatic labeling strategy is determined.

[0071] In some optional implementations, the calculation of the Pareto dominance relationship between any two of the automatic labeling strategies based on the labeling quality and the cost saving includes:

[0072] Establishing a labeling quality set and a cost saving set of the above automatic labeling strategy;

[0073] Obtain the number of the automatic labeling strategies in the automatic labeling strategy set;

[0074] Determine whether the number of the above strategies is less than or equal to the strategy number threshold;

[0075] If it is less than, traverse the above annotation quality set and the above cost saving set, and calculate the Pareto dominance relationship between any two of the above automatic annotation strategies.

[0076] In some optional implementations, the first automatic labeling strategy is determined based on the Pareto dominance relationship, including:

[0077] Calculate, according to the Pareto dominance relationship, a second number of remaining automatic labeling strategies dominated by each of the automatic labeling strategies, where the remaining automatic labeling strategies refer to the remaining automatic labeling strategies after any of the automatic labeling strategies is selected from the automatic labeling strategy set;

[0078] Prioritizing the automatic labeling strategies in the automatic labeling strategy set according to the second number, and determining the automatic labeling strategy with the highest second number as the first automatic labeling strategy;

[0079] If it is not less than, the automatic labeling strategy set is used as the decision space of the genetic algorithm, and the first automatic labeling strategy is obtained through the genetic algorithm.

[0080] In some optional implementations, the determining whether the first automatic labeling strategy satisfies the first quality assessment indicator includes:

[0081] Use the first automatic labeling strategy to label the quality inspection data;

[0082] Determine whether the first automatic labeling strategy is greater than a quality control threshold of a first quality assessment indicator according to the labeling result of the first automatic labeling strategy on the quality inspection data and the manual labeling result of the quality inspection data;

[0083] If so, determining that the first automatic labeling strategy satisfies the first quality assessment indicator;

[0084] If not, it is determined that the first automatic labeling strategy does not meet the first quality assessment indicator.

[0085] In some optional implementations, the determining that the first automatic labeling strategy does not meet the first quality assessment indicator further includes:

[0086] Determine the automatically labeled data of the above target labeling task as unlabeled data;

[0087] The following sampling and labeling operations are performed on the unlabeled data to update the first automatic labeling strategy until the Pareto objective value of the first automatic labeling strategy converges:

[0088] Random sampling is performed in the above unlabeled data to obtain random sampling data;

[0089] Calculating the contribution value of each data item in the random sampling data to the multi-objective optimization model, wherein the multi-objective optimization model is used to calculate the Pareto optimal target value of the first automatic labeling strategy;

[0090] According to the contribution value of each data item in the random sampling data, data items are selected from the random sampling data to form high contribution value sampling data;

[0091] Manually label the above high contribution value sampling data to obtain corresponding manual labeling results;

[0092] According to the above-mentioned high contribution value sampling data and the corresponding manual labeling results, the Pareto optimal target value of the above-mentioned first automatic labeling strategy is recalculated, the above-mentioned first automatic labeling strategy is updated based on the above-mentioned high contribution value sampling data and the corresponding manual labeling results, and the above-mentioned unlabeled data is updated according to the above-mentioned high contribution value sampling data and the corresponding manual labeling results, and the above-mentioned sampling and labeling operations are continued to be performed through the updated unlabeled data until the above-mentioned Pareto target value converges.

[0093] In a second aspect, the present disclosure provides a data labeling device, comprising:

[0094] A receiving unit, configured to receive a data annotation request, wherein the data annotation request includes at least one annotation task and a task type, annotation data, constraints, and annotation index values ​​corresponding to each annotation task, wherein the annotation data includes at least one data item, and the constraints include at least one constraint condition;

[0095] A determination unit, configured to determine a target labeling task according to the labeling index and the labeling index value of each labeling task;

[0096] A segmentation unit, used for segmenting the annotated data of the target annotating task into automatically annotated data and manually annotated data according to the constraints;

[0097] The determination unit is further used to determine the target automatic labeling strategy according to at least one of the task type, labeling quality and cost saving of the task to be labeled;

[0098] The labeling unit is used to label the above-mentioned automatic labeling data through the above-mentioned target automatic labeling strategy.

[0099] In a third aspect, the present disclosure provides an electronic device, including:

[0100] one or more processors;

[0101] a storage device having one or more programs stored thereon,

[0102] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any embodiment of the first aspect of the present disclosure.

[0103] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by one or more processors, implements the method described in any embodiment of the first aspect of the present disclosure.

[0104] The data annotation method, device, electronic device and storage medium provided by the embodiments of the present disclosure receive a data annotation request, wherein the data annotation request includes at least one annotation task and the task type, annotation data, constraint and annotation index value corresponding to each annotation task, the annotation data includes at least one data item, and the constraint includes at least one constraint condition. The target annotation task is determined according to the annotation index value of each annotation task. The annotation data of the target annotation task is divided into automatic annotation data and manual annotation data according to the constraint. The target automatic annotation strategy is determined according to at least one of the task type, annotation quality and cost saving of the task to be annotated. The automatic annotation data is annotated by the target automatic annotation strategy. By first determining the target annotation task, and then dividing the target annotation task into automatic annotation data and manual annotation data according to the constraint, further determining the target automatic annotation strategy according to at least one of the task type, annotation quality and cost saving of the task to be annotated, and annotating the automatic annotation data by the target automatic annotation strategy. By dynamically balancing multiple target values ​​such as data annotation quality, annotation efficiency and labor cost, better resource allocation and personalized data annotation configuration for each annotation task are achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] Other features, objects and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are only for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the drawings:

[0106] Figure 1 is a system architecture diagram of an embodiment of a data annotation system according to the present disclosure;

[0107] Figure 2 is a flow chart of an embodiment of a data annotation method according to the present disclosure;

[0108] Figure 3 is a structural schematic diagram of an embodiment of a data annotation device according to the present disclosure;

[0109] Figure 4 It is a structural diagram of a computer system of an electronic device suitable for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0110] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0111] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0112] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the data annotation method, apparatus, terminal device, and storage medium of the present disclosure can be applied.

[0113] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0114] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as data annotation applications, voice interaction applications, video conferencing applications, short video social applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0115] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with microphones and speakers, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Compression Standard Audio Layer 3), MP4 players (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Compression Standard Audio Layer 4), portable computers and desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, sending data annotation requests), or it can be implemented as a single software or software module. No specific limitation is made here.

[0116] The server 105 may be a server that provides various services, such as a background server that processes data annotation requests sent by the terminal devices 101, 102, and 103. The background server may perform corresponding processing based on the data annotation requests sent by the terminal devices.

[0117] In some cases, the data annotation request method provided by the present disclosure can be jointly performed by the terminal devices 101, 102, 103 and the server 105. For example, the step of "receiving a data annotation request" can be performed by the terminal devices 101, 102, 103, and the step of "determining a target annotation task according to the annotation index and the annotation index value of each annotation task" can be performed by the server 105. The present disclosure does not limit this. Accordingly, the data annotation device can also be respectively set in the terminal devices 101, 102, 103 and the server 105.

[0118] In some cases, the data labeling method provided in the present disclosure may be executed by the server 105 , and accordingly, the data labeling device may also be set in the server 105 . In this case, the system architecture 100 may not include the terminal devices 101 , 102 , and 103 .

[0119] In some cases, the data labeling method provided in the present disclosure may be executed by the terminal devices 101 , 102 , and 103 . Accordingly, the data labeling apparatus may also be disposed in the terminal devices 101 , 102 , and 103 . In this case, the system architecture 100 may not include the server 105 .

[0120] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0121] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0122] Continue to refer Figure 2 , Figure 2 A flowchart 200 of an embodiment of a data annotation method according to the present disclosure is shown. Figure 2 The data annotation method shown can be applied to Figure 1 The process 200 includes the following steps:

[0123] Step 201: receiving a data annotation request.

[0124] In this embodiment, the data labeling request may include at least one labeling task and the task type, labeling data, constraints and labeling index values ​​corresponding to each labeling task. The labeling data includes at least one data item, and the constraints include at least one constraint condition.

[0125] In this embodiment, the task type of the labeling task refers to how the labeling data needs to be labeled. For example, the task type of the labeling task can be to select the target object in the image using a rectangular box. The task type of the labeling task can also be to outline the target object in the image using an irregular polygon.

[0126] A data annotation request may include at least one annotation task. Different annotation tasks may correspond to the same annotation type, but may have different specific annotation targets. For example, both annotation task T1 and annotation task T2 are to select the target object in the image. Specifically, annotation task T1 is used to select the face image, and annotation task T2 is used to select the cat face image.

[0127] The labeled data refers to the data that needs to be labeled for the labeling task, and the labeled data includes at least one data item. For example, the labeled data may include at least one image that needs to be labeled.

[0128] Constraints are data annotation conditions predefined by the annotation demander based on the specific annotation task. For example, when the annotation task involves security issues such as user information and privacy, or when the user service accuracy is required to be relatively high, manual annotation is performed to ensure that the annotation service is safe and reliable. When the security requirements of the annotation task are lower, automatic annotation is used. Constraints include at least one constraint condition.

[0129] The labeling index value may include at least one of the number of unlabeled data, the expected time cost value of the labeling task, the estimated economic benefit value of the labeling task, etc. The specific value of the labeling index value is determined by the labeling demander. In addition, the labeling index value may be 0, indicating that the labeling index value is irrelevant to the labeling task.

[0130] Optionally, the labeling index value may be positively correlated with the importance of the labeling task. When there is a negative correlation between the labeling index value and the importance of the labeling task, the labeling index value may be reversely converted to form a new labeling index value that is positively correlated with the labeling task.

[0131] Step 202: determine the target labeling task according to the labeling index value of each labeling task.

[0132] This step is used to prioritize at least one labeling task in the labeling request, and then obtain the labeling task with the highest priority, that is, the target labeling task.

[0133] In some optional implementations, first, a first number of remaining labeling tasks dominated by each labeling task may be calculated according to the labeling index value of each labeling task. The remaining labeling tasks refer to the labeling tasks remaining after any labeling task is selected from at least one labeling task. Then, at least one labeling task is prioritized according to the first number, and the labeling task with the highest first number is determined as the target labeling task.

[0134] For example, the data annotation request received in step 201 includes N annotation tasks, that is, T = {T1, T2, ..., T N}, the labeled data corresponding to the N labeling tasks is D = {D1, D2, ..., D N}, where the labeling task T i The corresponding annotation data is D i , i is a positive integer between 1 and N, and the labeling task T i The corresponding annotation index is Labeling indicators

[0135] Here, n refers to the total number of labeled indicators.

[0136] For any two tasks T p , T q , where p, q∈[1,N], p≠q, if for the labeling index All indicators in T p With and T q equal and at least one better indicator value, then the labeling task T p Leading annotation task T q , that is, the labeling task T p Dominant, indicating that the labeling task T p Than the labeling task q important.

[0137] Calculate the first number of dominant positions for each labeling task, prioritize the labeling tasks according to the first number, and prioritize the labeling task with the highest first number T m Determine the target labeling task. m Priority marking is performed, where m is a positive integer between 1 and N.

[0138] In this step, the Pareto dominance relationship between the labeling tasks is used to determine the labeling order of the labeling tasks according to their importance.

[0139] Step 203 , dividing the labeled data of the target labeling task into automatically labeled data and manually labeled data according to the constraints.

[0140] This step is used to divide the labeled data of the target labeling task into automatic labeled data and manual labeled data after determining the target labeling task. The automatic labeled data is automatically labeled by the automatic labeling strategy, and the manual labeled data is manually labeled by humans. It should be noted that this step can also be implemented separately to divide the labeled data of any labeling task into automatic labeled data and manual labeled data, and is not limited to dividing the labeled data of the target labeling task into automatic labeled data and manual labeled data.

[0141] In this embodiment, it is determined whether each data item in the labeled data satisfies each constraint condition corresponding to the target labeling task. If so, the data item of the labeled data is determined as automatically labeled data; if not, the data item of the labeled data is determined as manually labeled data.

[0142] For example, for the target labeling task T m The corresponding annotation data D m , for the labeled data D m For each data item in, determine whether the data item satisfies all the conditions in constraint C, where C = {C1, C2, ..., C nc}, where nc refers to the labeling task T m The number of constraints in the corresponding constraint C. If the labeled data D m If data item 1 in constraint C satisfies all constraints, then data item 1 is determined as the automatically labeled data. If data item 2 does not satisfy all the constraints in constraint C, then data item 2 is determined as manually labeled data.

[0143] In this embodiment, for manually annotated data, further, since there are often some images with low visual similarity and inconsistent product categories in the manually annotated data, a pre-filtering strategy can be used to automatically remove some negative examples to find a solution that balances prediction accuracy and cost savings.

[0144] First, the target pre-filtering strategy is determined according to the task type of the target labeling task.

[0145] Specifically, a pre-filtering strategy set is obtained, the pre-filtering strategy set includes at least one pre-filtering strategy, each pre-filtering strategy corresponds to a task type of a filterable annotation task, and the pre-filtering strategy is used to discard negative examples in the manually annotated data. Among them, negative examples refer to some images with low visual similarity and inconsistent product categories in the manually annotated data.

[0146] The task type of the target labeling task is matched with the task type of the filterable labeling task corresponding to the pre-filtering strategy, and it is determined whether the task type of the target labeling task exists in the task type of the filterable labeling task corresponding to the pre-filtering strategy. If so, the pre-filtering strategy corresponding to the task type of the target labeling task is determined as the target pre-filtering strategy. In this embodiment, the pre-filtering strategy set can be a preset pre-filtering strategy set.

[0147] Then, according to the target pre-filtering strategy, it is determined whether each data item in the manually labeled data meets the target pre-filtering strategy threshold. If it meets the threshold, the data item is sent to the manually labeled data item queue, and a labeling terminal is assigned to the data item in the manually labeled data item queue. The labeler labels the data item through the labeling terminal. If it does not meet the threshold, the data item is determined as a negative example and discarded.

[0148] Step 204 : determining a target automatic labeling strategy according to at least one of the task type, labeling quality, and cost saving of the task to be labeled.

[0149] In this embodiment, the task to be labeled refers to the unlabeled labeling task in the data labeling request, the labeling quality refers to the quality of data labeling, and the cost saving refers to the cost that can be saved by labeling the labeling task, which may include at least one of labor cost, time cost, and expense cost.

[0150] Specifically, it is determined whether there is a historical automatic labeling strategy corresponding to the target labeling task according to the task type of the target labeling task.

[0151] In some optional implementations, first, a historical automatic annotation strategy set is obtained. The historical automatic annotation strategy set can be directly obtained from a database. The historical automatic annotation strategy set includes at least one historical automatic annotation strategy, each historical automatic annotation strategy corresponds to a task type of an annotable task, and the historical automatic annotation strategy is used to annotate the automatically annotated data.

[0152] Then, the task type of the target labeling task is matched with the task type of the labelable task corresponding to the historical automatic labeling strategy.

[0153] According to whether there is a task type of the target labeling task among the task types of the labelable tasks corresponding to the historical automatic labeling strategy, it is determined whether there is a historical automatic labeling strategy corresponding to the target labeling task.

[0154] If it is determined that there is no historical automatic labeling strategy corresponding to a labelable task whose task type is the same as the task type of the automatic labeling data, it is determined that there is no historical automatic labeling strategy corresponding to the target labeling task.

[0155] In some optional implementations, if it is determined that there is no historical automatic labeling strategy corresponding to the target labeling task, the automatically labeled data is determined to be manually labeled data, and the automatically labeled data is labeled manually.

[0156] If it is determined that there exists a task type of a labelable task corresponding to a historical automatic labeling strategy that is the same as the task type of the automatic labeling data, then it is determined that there exists a historical automatic labeling strategy corresponding to the target labeling task.

[0157] In some optional implementations, when it is determined that there is a historical automatic labeling strategy corresponding to the target labeling task, the historical automatic labeling strategy is determined as the target historical automatic labeling strategy, and it is determined whether the target historical automatic labeling strategy meets the first quality evaluation indicator.

[0158] It is determined whether the target history automatic labeling strategy meets the first quality assessment indicator, that is, it is determined whether the target history automatic labeling strategy can meet the labeling requirements for the automatic labeling data.

[0159] In some optional implementations, whether the target history automatic labeling strategy meets the first quality evaluation indicator may be determined in the following manner.

[0160] First, randomly select data with a preset quality inspection ratio from the automatically labeled data, and determine the data with the preset quality inspection ratio as quality inspection data. The quality inspection data is data for judging whether the target history automatic labeling strategy can meet the labeling requirements of the automatically labeled data.

[0161] Then, the target history automatic labeling strategy is used to label the quality inspection data, and the labeling results of the quality inspection data by the target history automatic labeling strategy and the manual labeling results of the quality inspection data are obtained.

[0162] Finally, it is determined whether the target history automatic labeling strategy meets the first quality assessment indicator based on the labeling results of the target history automatic labeling strategy on the quality inspection data and the manual labeling results on the quality inspection data.

[0163] For example, in the historical automatic annotation strategy set, the target historical automatic annotation strategy is determined to be S hist , in automatic data annotation Randomly select the data with RQA ratio (i.e., preset quality inspection ratio) as the quality inspection data Generally, rqa can be 5% or other proportions, which is not specifically limited here.

[0164] Quality inspection data through target history automatic annotation strategy Perform labeling to obtain the labeling results of the target history automatic labeling strategy Manual quality inspection of data Perform annotation to obtain annotation results

[0165] In this embodiment, the quality inspection data is automatically annotated and manually annotated by the target history automatic annotation strategy and manually, and then it is judged whether the target history automatic annotation strategy meets the first quality evaluation index according to the results of automatic annotation and manual annotation. The first quality evaluation index includes at least one of the following: recall rate, precision rate, accuracy rate, false positive rate and false negative rate.

[0166] Recall rate = true positive / (true positive + false negative), precision rate = true positive / (true positive + false positive), accuracy rate = (true positive + true negative) / total quality inspection data, false positive rate = false positive / (false positive + true negative), false negative rate = false negative / (false negative + true positive).

[0167] Among them, true positives indicate the number of data items whose historical automatic labeling strategy predicts the target as positive examples, and whose manual labeling results are also positive examples; false negatives indicate the number of data items whose historical automatic labeling strategy predicts the target as negative examples, and whose manual labeling results are also positive examples; false positives indicate the number of data items whose historical automatic labeling strategy predicts the target as positive examples, and whose manual labeling results are also negative examples; true negatives indicate the number of data items whose historical automatic labeling strategy predicts the target as negative examples, and whose manual labeling results are also negative examples.

[0168] In some optional implementations, it is determined whether the target history automatic labeling strategy meets the first quality assessment indicator based on the labeling results of the target history automatic labeling strategy on the quality inspection data and the manual labeling results on the quality inspection data.

[0169] Specifically, the recall rate, precision rate, accuracy rate, false positive rate and false negative rate are calculated based on the labeling results of the quality inspection data by the target history automatic labeling strategy and the manual labeling results of the quality inspection data. It is judged whether the target history automatic labeling strategy meets the first quality evaluation indicator based on whether the recall rate, precision rate, accuracy rate, false positive rate and false negative rate are greater than the recall rate threshold, precision rate threshold, accuracy rate threshold, false positive rate threshold and false negative rate threshold respectively.

[0170] If the recall rate, precision rate, accuracy rate, false positive rate and false negative rate are respectively greater than the recall rate threshold, the precision rate threshold, the accuracy rate threshold, the false positive rate threshold and the false negative rate threshold, then it is determined that the target history automatic labeling strategy meets the first quality evaluation indicator.

[0171] If at least one of the recall rate, precision rate, accuracy rate, false positive rate and false negative rate is not greater than the recall rate threshold, the precision rate threshold, the accuracy rate threshold, the false positive rate threshold and the false negative rate threshold, it is determined that the target history automatic labeling strategy does not meet the first quality evaluation indicator.

[0172] For example, the recall threshold ηrecall , precision threshold η pr , accuracy threshold η acc , false positive rate threshold η fnr and the false negative rate threshold η fpr The values ​​may be 95%, 95%, 98%, 4%, and 4% respectively. These values ​​are only exemplary and are not limited to specific values.

[0173] When the recall rate, precision rate, accuracy rate, false positive rate and false negative rate of the target history automatic labeling strategy for the quality inspection data are greater than 95%, 95%, 98%, 4% and 4% respectively, it is determined that the target history automatic labeling strategy meets the labeling requirements for the automatically labeled data, that is, the target history automatic labeling strategy meets the first quality evaluation indicator.

[0174] When the target history automatic labeling strategy meets the first quality evaluation index, all the automatically labeled data are labeled using the target history automatic labeling strategy.

[0175] In some optional implementations, when the target history automatic labeling strategy does not meet the first quality assessment indicator, for example, when the target history automatic labeling strategy has at least one of the recall rate, precision rate, accuracy rate, false positive rate and false negative rate of the quality inspection data that is not greater than η recall , η pr , η acc , η fnr , η fpr When, for example, the recall rate is less than 95%, the target history automatic labeling strategy does not meet the first quality evaluation indicator.

[0176] The automatic labeling strategy set is acquired according to the task type of the target labeling task. Here, the automatic labeling strategy set includes at least one automatic labeling strategy.

[0177] In some optional implementations, the automatic labeling strategy set may be obtained in the following manner: specifically, first, according to the task type of the target labeling task, X feature extraction models and Y classifier models are selected from the predefined feature extraction model library and classifier model library, respectively, where X and Y are both positive integers. Then, any feature extraction model in the X feature extraction models and any classifier model in the Y classifier models are arranged and combined to obtain the automatic labeling strategy set.

[0178] Among them, the feature extraction model is used to extract data features of automatically labeled data, and the classifier model is used to classify the extracted data features.

[0179] It should be noted that when two feature extraction models or two classifier models differ only in model parameters, the two feature extraction models or classifier models with different parameters are regarded as different feature extraction models or classifier models. In addition, different task types can have different feature extraction model libraries and classifier model libraries. For example, images have feature extraction model libraries and classifier model libraries corresponding to images, and texts have feature extraction model libraries and classifier model libraries corresponding to texts.

[0180] Any feature extraction model among the X feature extraction models and any classifier model among the Y classifier models are arranged and combined to obtain an automatic labeling strategy set.

[0181] For example, the feature extraction models are F1, F2, ..., F X , the classifier models are L1, L2, ...L Y , X and Y are the number of feature extraction models and the number of classifier models respectively. X , respectively, and the classifier models L1, L2, ...L Y , the classifier models can be arranged and combined in any way to obtain the automatic labeling strategy set S, S = S1, S2, S3, ..., S N’ , where N' is the number of automatic labeling strategies in the set S. In theory, N' can be the product of X and Y.

[0182] After acquiring the automatic labeling strategy set, a target automatic labeling strategy is determined based on at least one automatic labeling strategy.

[0183] In some optional implementations, the annotation quality and cost savings of each automatic annotation strategy are calculated.

[0184] In this embodiment, in order to measure the labeling quality and cost savings, Pareto is used to take the optimal labeling quality and cost savings as the goals, and the quality control constraints and labeling cost constraints as constraints to seek the optimal solution between labeling quality and cost savings, and the first automatic labeling strategy is selected to label the automatic labeling data.

[0185] In some optional implementations, the annotation quality of each automatic annotation strategy may be calculated in the following manner.

[0186] Randomly select quality inspection data, use each automatic labeling strategy in the above automatic labeling strategy set to label the quality inspection data, and obtain the labeling results of each automatic labeling strategy on the quality inspection data. In addition, manually label the quality inspection data to obtain the manual labeling results, and then calculate the labeling quality of each automatic labeling strategy based on the labeling results of each automatic labeling strategy on the quality inspection data and the manual labeling results of the quality inspection data.

[0187] In some optional implementations, the annotation quality of each automatic annotation strategy is calculated, and the recall rate, precision rate, accuracy rate and correct identification negative example ratio of each automatic annotation strategy can be calculated respectively according to the annotation results of each automatic annotation strategy on the quality inspection data and the manual annotation results of the quality inspection data.

[0188] The recall rate, precision rate, accuracy rate and the proportion of correctly identified negative examples are linearly weighted to obtain the annotation quality of each automatic annotation strategy.

[0189] For example, for any automatic labeling strategy S in the automatic labeling strategy set j , j∈[1,N'], using the automatic labeling strategy S j Label the quality inspection data to obtain automatic labeling results Based on the automatic labeling strategy S j Automatic annotation results And the manual annotation results The recall rate, precision rate, accuracy rate and the proportion of correctly identified negative examples are calculated respectively, where the proportion of correctly identified negative examples = true negatives / (true negatives + false positives).

[0190] Furthermore, the quality evaluation index Q is calculated based on the recall rate, precision rate, accuracy rate and the proportion of correctly identified negative examples. j , j∈[1,N'], quality assessment index Q j Used to represent the automatic labeling strategy S j The quality of annotation.

[0191] Q(.)=ω1·Recall+ω2·Pricision+ω3·Accuracy+ω4·Drop neg

[0192] Q j =Q(S j )

[0193] Among them, ω1, ω2, ω3, and ω4 represent the weights corresponding to the recall rate, precision rate, accuracy rate, and the proportion of negative examples correctly identified, respectively. Recall represents the recall rate, Pricision represents the precision rate, Accuracy represents the accuracy rate, and Drop represents the weights corresponding to the proportion of negative examples correctly identified. neg It represents the proportion of negative examples that are correctly identified, Q(.) is the quality evaluation function, and the annotation quality of the automatic annotation strategy can be obtained based on the quality evaluation function.

[0194] In some optional implementations, the cost savings of each automatic annotation strategy may be calculated in the following manner.

[0195] The task characteristics of the target labeling task are obtained, including the number of data items of the automatically labeled data and the average cost of manually labeling a single data item of the automatically labeled data. The cost savings of each automatic labeling strategy are calculated based on the task characteristics and the labeling results of each automatic labeling strategy on the quality inspection data.

[0196] For example, the cost saving function S can be used cost (·) Calculate the cost savings of each automatic labeling strategy. Where:

[0197] S cost (·)=L·P·Accuracy

[0198]

[0199] Where L is the number of data items in the automatically labeled data, P is the average cost of manually labeling a single data item in the automatically labeled data, and S cost (·) is the cost saving function, based on which the cost saving of the automatic labeling strategy is obtained.

[0200] In some optional implementations, after the annotation quality and the cost saving are obtained, a first automatic annotation strategy is determined according to the annotation quality and the cost saving.

[0201] This step is used to select the first automatic labeling strategy from all automatic labeling strategies according to the labeling quality and the cost saving after obtaining the labeling quality and the cost saving of each automatic labeling strategy.

[0202] That is, a Pareto multi-objective optimization model is constructed to evaluate the automatic labeling strategy. The Pareto multi-objective optimization model is expressed as follows:

[0203] maxQ j =Q(S j )

[0204]

[0205] Q lb ≤Q j

[0206]

[0207] j∈[1,N']

[0208] Among them, Q lb is the lower limit of the annotation quality, is the lower limit of cost saving. The optimal solution between labeling quality and cost saving can be found through the above Pareto multi-objective optimization model.

[0209] In some optional implementations, based on the annotation quality and the cost saving, the Pareto dominance relationship between any two automatic annotation strategies is calculated, and based on the Pareto dominance relationship, the first automatic annotation strategy is determined.

[0210] First, establish the annotation quality set and cost saving set of the automatic annotation strategy. That is, the annotation quality of each automatic annotation strategy constitutes the annotation quality set Q and the cost saving set S of each automatic annotation strategy constitutes the cost saving set cost , Then, the number of strategies of the automatic labeling strategies in the automatic labeling strategy set is obtained, and it is determined whether the number of strategies is less than or equal to the strategy number threshold.

[0211] If it is less than, then traverse the labeling quality set and the cost saving set, calculate the Pareto dominance relationship between any two automatic labeling strategies, and calculate the second quantity of each automatic labeling strategy dominating the remaining automatic labeling strategies based on the Pareto dominance relationship. The remaining automatic labeling strategies refer to the automatic labeling strategies remaining after selecting any automatic labeling strategy in the automatic labeling strategy set. The automatic labeling strategies in the automatic labeling strategy set are prioritized according to the second quantity, and the automatic labeling strategy with the highest second quantity is determined as the first automatic labeling strategy.

[0212] If it is not less than, it means that the decision variable space of the automatic labeling strategy is relatively large and is not suitable for traversal and solution. In this case, the automatic labeling strategy set can be used as the decision space of the genetic algorithm, and the first automatic labeling strategy is obtained through the genetic algorithm.

[0213] In this embodiment, the genetic algorithm may be the NSGA2 algorithm or the NSPA algorithm.

[0214] The following uses the NSGA2 algorithm as an example:

[0215] Step 1: Get the initial population.

[0216] Some automatic labeling strategies are randomly selected from the automatic labeling strategy set to form an initialization automatic labeling strategy, that is, the initialization population.

[0217] Step 2: Calculate individual fitness.

[0218] Calculate the individual fitness of each automatic labeling strategy in the initialization population. Individual fitness is an indicator of the automatic labeling strategy's quality and cost savings.

[0219] Step 3: Calculate the crowding distance value.

[0220] For each automatic labeling strategy, calculate its crowding distance value in different automatic labeling strategies. Crowding distance refers to the density around the automatic labeling strategy, which can be used to measure the diversity of solutions. By calculating the crowding distance value of the automatic labeling strategy, the automatic labeling strategy can maintain a certain degree of heterogeneity during the iteration process. For example, automatic labeling strategies A and B both dominate 5 solutions. In order to maintain the difference in the next iteration, the automatic labeling strategy with a larger crowding distance value is selected for iteration. The smaller the crowding distance value, the more homogeneous the automatic labeling strategy.

[0221] Step 4: Perform non-dominated sorting on the automatic labeling strategies.

[0222] According to the labeling quality and cost saving of the automatic labeling strategy, the non-dominated relationship between any two automatic labeling strategies is obtained, and then the automatic labeling strategies are non-dominated sorted. Non-dominated sorting means that the automatic labeling strategies are divided into multiple levels. There is no dominance relationship between the automatic labeling strategies in each level, that is, no automatic labeling strategy can outperform another automatic labeling strategy in all objective functions (i.e., labeling quality and cost saving).

[0223] Step 5: Select the operation.

[0224] After the above non-dominated sorting of the automatic labeling strategies, a certain number of automatic labeling strategies are further selected from the automatic labeling strategies to join the next generation population, that is, the next generation automatic labeling strategies. For example, in order to maintain the diversity of the next generation population, the tournament method can be used to select the automatic labeling strategies.

[0225] Step 6: Crossover operation.

[0226] The selected automatic labeling strategy is cross-operated to generate a new individual, that is, a new automatic labeling strategy. The cross-operation can be performed in a variety of ways, such as a single-point cross-operation or a double-point cross-operation. In this embodiment, the cross-operation of the automatic labeling strategy refers to the cross-operation of the binary code corresponding to each automatic labeling strategy, and the binary code of each automatic labeling strategy is loaded together to form the cross of the automatic labeling strategy.

[0227] Step 7: Mutation operation.

[0228] After the automatic labeling strategy is cross-operated, the cross-operated automatic labeling strategy is mutated to further increase the diversity of the automatic labeling strategy. The mutation operation of the automatic labeling strategy refers to mutating the binary code of the automatic labeling strategy.

[0229] Through the above process, the next generation of population, i.e., the next generation of automatic labeling strategy, is generated based on the selection, crossover, mutation, and mutation operations of the initial population. The next generation of population includes some automatic labeling strategies from the previous generation of population and some newly generated automatic labeling strategies. Then repeat the above steps 2-7 until the preset number of iterations is reached or the termination condition is met, and the genetic algorithm outputs the first automatic labeling strategy.

[0230] The Pareto dominance relationship between any two automatic labeling strategies is calculated, and the highest automatic labeling strategy is determined as the first automatic labeling strategy.

[0231] In some optional implementations, it is determined whether the first automatic labeling strategy satisfies a first quality assessment indicator.

[0232] In this embodiment, the quality inspection data is automatically annotated and manually annotated by the first automatic annotation strategy, and then it is judged whether the first automatic annotation strategy meets the first quality assessment indicator according to the annotation results of the first automatic annotation strategy and the manual annotation results. Specifically, the quality inspection data is annotated using the first automatic annotation strategy, and then it is judged whether the first automatic annotation strategy is greater than the quality control threshold of the first quality assessment indicator according to the annotation results of the quality inspection data by the first automatic annotation strategy and the manual annotation results of the quality inspection data; if it is greater, it is determined that the first automatic annotation strategy meets the first quality assessment indicator, and the first automatic annotation strategy is determined as the target automatic annotation strategy. If it is not greater, it is determined that the first automatic annotation strategy does not meet the first quality assessment indicator.

[0233] When the first automatic labeling strategy does not meet the first quality evaluation index, the automatically labeled data is further divided into unlabeled data and labeled data. At this time, the unlabeled data is equal to the automatically labeled data, and the labeled data is an empty set.

[0234] First, the automatically labeled data of the target labeling task is determined as unlabeled data, and the following sampling and labeling operations are performed on the unlabeled data to update the first automatic labeling strategy until the Pareto objective value of the first automatic labeling strategy converges.

[0235] Specifically, random sampling is performed in the unlabeled data to obtain random sampling data, and the contribution value of each data item in the random sampling data to the multi-objective optimization model is calculated. The multi-objective optimization model is used to calculate the Pareto optimal target value of the first automatic labeling strategy. According to the contribution value of each data item in the random sampling data, data items are selected in the random sampling data to form high-contribution value sampling data. The high-contribution value sampling data is manually labeled to obtain the corresponding manual labeling results. According to the high-contribution value sampling data and the corresponding manual labeling results, the Pareto optimal target value of the first automatic labeling strategy is recalculated. The first automatic labeling strategy is updated based on the high-contribution value sampling data and the corresponding manual labeling results, and the unlabeled data is updated according to the high-contribution value sampling data and the corresponding manual labeling results. The sampling and labeling operations are continued through the updated unlabeled data until the Pareto target value converges.

[0236] It should be noted that the multi-objective optimization model here may be the above-mentioned Pareto multi-objective optimization model. In an optional embodiment, the contribution value of each data item may be the contribution value of the sampled data to the multi-objective optimization model determined according to the density weight or variance of the sampled data.

[0237] Data items are selected from the random sampling data to form high contribution value sampling data, wherein the high contribution value sampling data may be data items whose contribution value is greater than a preset contribution value threshold.

[0238] For example, the larger the variance of the sampled data, the greater the contribution of the sampled data to the multi-objective optimization model. Then, the data items with variance values ​​greater than the preset contribution value threshold are determined as high-contribution value sampled data.

[0239] According to the high contribution value sampling data and the corresponding manual labeling results, the Pareto optimal target value of the first automatic labeling strategy is recalculated, and the first automatic labeling strategy is updated based on the high contribution value sampling data and the corresponding manual labeling results.

[0240] In this embodiment, calculating the Pareto optimal target value refers to calculating the labeling quality and cost saving. After manually labeling the high contribution value sampled data, the high contribution value sampled data is stored as the above-mentioned labeled data, and the unlabeled data is updated. The updated unlabeled data does not include the manually labeled high contribution value sampled data. Then, the sampling and labeling operations are continued through the updated unlabeled data until the Pareto target value converges.

[0241] Pareto objective value convergence means that the change of the Pareto optimal objective value between two adjacent iterations is less than the convergence threshold Δ1, Δ2, where Δ1 refers to the convergence threshold of annotation quality, and Δ2 refers to the convergence threshold of cost saving. Generally, the convergence thresholds Δ1, Δ2 are data between 0 and 1, and may not be data between 0 and 1. The convergence thresholds Δ1, Δ2 can be dynamically adjusted. In addition, the smaller the convergence threshold, the better, indicating that the annotation of the multi-objective optimization model, i.e., the first automatic annotation strategy, is more accurate.

[0242] In addition, the optimized first automatic labeling strategy may be stored in a database, and when a labeling task matches the first automatic labeling strategy, the first automatic labeling strategy is used to label the automatic labeling data of the labeling task.

[0243] Step 205: label the automatically labeled data using the target automatic labeling strategy.

[0244] When it is determined that the first automatic labeling strategy satisfies the first quality assessment indicator, the first automatic labeling strategy is determined to be a target automatic labeling strategy, and the automatically labeled data is labeled using the target automatic labeling strategy.

[0245] The data labeling method provided by the embodiment of the present disclosure receives a data labeling request, then determines the labeling priority of the labeling task through the labeling indicators and indicator values ​​in the data labeling request, and then divides the labeled data into automatically labeled data and manually labeled data through constraints. For the automatically labeled data, first, determine whether to label the automatically labeled data through the historical labeling strategy through the labeling quality of the historical automatic labeling strategy. If not, obtain the automatic labeling strategy again, determine the first automatic labeling strategy based on the labeling quality and cost saving of the automatic labeling strategy, and then judge the labeling quality of the first automatic labeling strategy to determine whether to label the automatically labeled data through the first automatic labeling strategy. In the present disclosure, better resource allocation and personalized labeling configuration for each labeling task are achieved by dynamically balancing multiple target values ​​such as data labeling quality, labeling efficiency, and labor cost.

[0246] Further references Figure 3 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a data labeling device. Figure 2 Corresponding to the method embodiment shown, the apparatus can be specifically applied to various terminal devices.

[0247] like Figure 3 As shown, the data labeling device of this embodiment, the device 300 includes: a receiving unit 301, a determining unit 302, a segmenting unit 303, a labeling unit 304, a judging unit 305, an acquiring unit 306, a calculating unit 307 and a processing unit 308.

[0248] In this embodiment, the specific processing of the receiving unit 301, the determining unit 302, the segmenting unit 303, the labeling unit 304, the judging unit 305, the acquiring unit 306, the calculating unit 307 and the processing unit 308 of the data labeling device and the technical effects thereof can be referred to respectively. Figure 2 The relevant descriptions of steps 201 to 205 in the corresponding embodiment are not repeated here.

[0249] The receiving unit 301 is used to receive a data annotation request, wherein the data annotation request includes at least one annotation task and a task type, annotation data, constraints, and annotation index values ​​corresponding to each annotation task, wherein the annotation data includes at least one data item, and the constraints include at least one constraint condition;

[0250] A determination unit 302, configured to determine a target labeling task according to the labeling indicator value of each labeling task;

[0251] A segmentation unit 303, configured to segment the annotated data of the target annotating task into automatically annotated data and manually annotated data according to the constraints;

[0252] The determination unit 302 is further used to determine the target automatic labeling strategy according to at least one of the task type, labeling quality and cost saving of the task to be labeled;

[0253] The labeling unit 304 is used to label the automatically labeled data using a target automatic labeling strategy.

[0254] In some optional implementations, the determination unit 302 is further configured to determine the target automatic labeling strategy according to at least one of the task type, labeling quality and cost saving of the task to be labeled, including:

[0255] The determination unit 302 is further configured to determine whether there is a historical automatic labeling strategy corresponding to the target labeling task according to the task type of the target labeling task;

[0256] If so, the determination unit 305 is configured to determine the above-mentioned historical automatic labeling strategy as the target historical automatic labeling strategy, and determine whether the above-mentioned target historical automatic labeling strategy meets the first quality assessment indicator;

[0257] If not, an acquiring unit 306 is configured to acquire an automatic labeling strategy set according to the task type of the target labeling task, wherein the automatic labeling strategy set includes at least one automatic labeling strategy;

[0258] The determination unit 302 is further configured to determine a target automatic labeling strategy based on at least one automatic labeling strategy.

[0259] In some optional implementations, the determination unit 302 is further configured to determine a target automatic labeling strategy based on at least one automatic labeling strategy.

[0260] A calculation unit 307, used to calculate the annotation quality and cost saving of each automatic annotation strategy;

[0261] The determining unit 302 is further configured to determine a first automatic labeling strategy according to the labeling quality and the cost saving;

[0262] The judging unit 305 is further configured to judge whether the first automatic labeling strategy satisfies a first quality assessment indicator;

[0263] If satisfied, the determining unit 302 is further configured to determine the first automatic labeling strategy as the target automatic labeling strategy.

[0264] In some optional implementations, the determining unit 302 is further configured to determine a target labeling task according to the labeling indicator value of each labeling task, including:

[0265] The calculation unit 307 is further used to calculate the first number of remaining labeling tasks dominated by each labeling task according to the labeling indicator value of each labeling task, where the remaining labeling tasks refer to the labeling tasks remaining after any labeling task is selected from the at least one labeling task;

[0266] The determining unit 302 is further configured to prioritize the at least one labeling task according to the first number, and determine the labeling task with the highest first number as the target labeling task.

[0267] In some optional implementations, the segmentation unit 303 is further configured to segment the labeled data of the target labeling task into automatically labeled data and manually labeled data according to the constraints, including:

[0268] The judgment unit 305 is further used to judge whether each data item in the labeled data satisfies each constraint condition of the constraint corresponding to the target labeling task;

[0269] If satisfied, the determination unit 302 is further configured to determine the data item of the annotated data as automatically annotated data;

[0270] If not, the determination unit 302 is further configured to determine the data item of the annotated data as manually annotated data.

[0271] In some optional implementations, the determining unit 302 is further configured to determine the data item of the annotated data as manually annotated data, and further includes:

[0272] The determination unit 302 is further configured to determine a target pre-filtering strategy according to the task type of the target labeling task;

[0273] The judgment unit 305 is further configured to judge whether each data item in the manually labeled data satisfies the target pre-filtering strategy threshold value according to the target pre-filtering strategy;

[0274] If satisfied, the processing unit 308 is used to send the data item to a manually labeled data item queue;

[0275] If not, the determination unit 302 is further configured to determine the data item as a negative example and discard it.

[0276] In some optional implementations, the determination unit 302 is further configured to determine a target pre-filtering strategy according to a task type of the target labeling task, including:

[0277] The acquisition unit 306 is further used to acquire a pre-filtering strategy set, the pre-filtering strategy set includes at least one pre-filtering strategy, each of the pre-filtering strategies corresponds to a task type that can filter the labeling task, and the pre-filtering strategy is used to discard negative examples in the manually labeled data;

[0278] The processing unit 308 is further configured to match the task type of the target labeling task with the task type of the filterable labeling task corresponding to the pre-filtering strategy, and the determining unit 302 is further configured to determine whether the task type of the target labeling task exists in the task type of the filterable labeling task corresponding to the pre-filtering strategy;

[0279] If so, the determination unit 302 is further configured to determine the pre-filtering strategy corresponding to the task type of the target labeling task as the target pre-filtering strategy.

[0280] In some optional implementations, the determining unit 302 is further configured to determine whether there is a historical automatic labeling strategy corresponding to the target labeling task according to the task type of the target labeling task, including:

[0281] The acquisition unit 306 is further used to acquire a historical automatic annotation strategy set, wherein the historical automatic annotation strategy set includes at least one historical automatic annotation strategy, each of which corresponds to a task type that can be annotated; the historical automatic annotation strategy is used to annotate the automatically annotated data;

[0282] The processing unit 308 is further configured to match the task type of the target labeling task with the task type of the labelable task corresponding to the historical automatic labeling strategy;

[0283] The determination unit 302 is further configured to determine whether there is the historical automatic labeling strategy corresponding to the target labeling task according to whether there is a task type of the target labeling task among the task types of labelable tasks corresponding to the historical automatic labeling strategy.

[0284] In some optional implementations, the determination unit 302 is further configured to determine whether there is the historical automatic labeling strategy corresponding to the target labeling task according to whether there is a task type of the target labeling task in the task types of the labelable tasks corresponding to the historical automatic labeling strategy, and further includes:

[0285] If not, the determining unit 302 is further configured to determine the automatically labeled data as manually labeled data.

[0286] In some optional implementations, the judgment unit 305 is further configured to judge whether the target history automatic labeling strategy meets the first quality assessment indicator, including:

[0287] The processing unit 308 is further used to randomly extract data with a preset quality inspection ratio from the automatically labeled data, and the determining unit 302 is further used to determine the data with the preset quality inspection ratio as the quality inspection data;

[0288] The above-mentioned labeling unit 304 is further used to label the above-mentioned quality inspection data using the above-mentioned target history automatic labeling strategy;

[0289] The acquisition unit 306 is further used to acquire the labeling result of the target history automatic labeling strategy on the quality inspection data and the manual labeling result of the quality inspection data;

[0290] The judgment unit 305 is further configured to judge whether the target history automatic labeling strategy meets the first quality assessment indicator according to the labeling result of the quality inspection data by the target history automatic labeling strategy and the manual labeling result of the quality inspection data.

[0291] In some optional implementations, the first quality assessment indicator includes recall rate, precision rate, accuracy rate, false positive rate and false negative rate; and

[0292] The judgment unit 305 is further configured to judge whether the target history automatic labeling strategy meets the first quality assessment indicator according to the labeling result of the quality inspection data by the target history automatic labeling strategy and the manual labeling result of the quality inspection data, including:

[0293] The calculation unit 307 is further used to calculate the recall rate, the precision rate, the accuracy rate, the false positive rate and the false negative rate according to the labeling results of the quality inspection data by the target history automatic labeling strategy and the manual labeling results of the quality inspection data;

[0294] The judgment unit 305 is further used to judge whether the target history automatic labeling strategy meets the first quality evaluation indicator according to whether the recall rate, the precision rate, the accuracy rate, the false positive rate and the false negative rate are respectively greater than the recall rate threshold, the precision rate threshold, the accuracy rate threshold, the false positive rate threshold and the false negative rate threshold;

[0295] The determination unit 302 is further configured to determine that the target history automatic labeling strategy meets the first quality assessment indicator if the recall rate, the precision rate, the accuracy rate, the false positive rate, and the false negative rate are respectively greater than the recall rate threshold, the precision rate threshold, the accuracy rate threshold, the false positive rate threshold, and the false negative rate threshold;

[0296] The determination unit 302 is also used to determine that the target history automatic labeling strategy does not meet the first quality assessment indicator if at least one of the recall rate, the precision rate, the accuracy rate, the false positive rate and the false negative rate is not greater than the recall rate threshold, the precision rate threshold, the accuracy rate threshold, the false positive rate threshold and the false negative rate threshold.

[0297] In some optional implementations, the judgment unit 305 is further configured to judge whether the target history automatic labeling strategy meets the first quality assessment indicator, and further includes:

[0298] If satisfied, the labeling unit 304 is further configured to label the automatically labeled data using the target history automatic labeling strategy.

[0299] In some optional implementations, the acquisition unit 306 is further configured to acquire the automatic labeling strategy set according to the task type of the target labeling task, including:

[0300] The processing unit 308 is further configured to select X feature extraction models and Y classifier models from the predefined feature extraction model library and classifier model library respectively according to the task type of the target labeling task, where X and Y are both positive integers;

[0301] The processing unit 308 is further configured to arrange and combine any feature extraction model among the X feature extraction models and any classifier model among the Y classifier models to obtain an automatic labeling strategy set.

[0302] In some optional implementations, the calculation unit 308 is further used to calculate the annotation quality and cost saving of each automatic annotation strategy, including:

[0303] The above-mentioned labeling unit 304 is further used to label the above-mentioned quality inspection data using each of the above-mentioned automatic labeling strategies respectively;

[0304] The calculation unit 307 is further used to calculate the labeling quality of each of the automatic labeling strategies according to the labeling results of the quality inspection data by each of the automatic labeling strategies and the manual labeling results of the quality inspection data;

[0305] The acquisition unit 306 is further used to acquire the task characteristics of the target labeling task, wherein the task characteristics include the number of data items of the automatically labeled data and the average cost of manually labeling a single data item of the automatically labeled data;

[0306] The calculation unit 307 is further configured to calculate the cost savings of each of the automatic labeling strategies according to the task characteristics and the labeling results of the quality inspection data by each of the automatic labeling strategies.

[0307] In some optional implementations, the calculation unit 307 is further configured to calculate the annotation quality of each automatic annotation strategy according to the annotation results of the quality inspection data by each automatic annotation strategy and the manual annotation results of the quality inspection data, including:

[0308] The calculation unit 307 is further used to calculate the recall rate, precision rate, accuracy rate and correct negative example ratio of each automatic labeling strategy according to the labeling results of the quality inspection data by each automatic labeling strategy and the manual labeling results of the quality inspection data.

[0309] The processing unit 308 is further configured to linearly weight the recall rate, the precision rate, the accuracy rate and the proportion of correctly identified negative examples to obtain the labeling quality of each of the automatic labeling strategies.

[0310] In some optional implementations, the determining unit 302 is further configured to determine the first automatic labeling strategy according to the labeling quality and the cost saving, including:

[0311] The processing unit 308 is further configured to calculate a Pareto dominance relationship between any two of the automatic labeling strategies based on the labeling quality and the cost saving.

[0312] The determination unit 302 is further configured to determine a first automatic labeling strategy based on the Pareto dominance relationship.

[0313] In some optional implementations, the calculation unit 307 is further configured to calculate the Pareto dominance relationship between any two of the automatic labeling strategies based on the labeling quality and the cost saving, including:

[0314] The processing unit 308 is further used to establish a labeling quality set and a cost saving set of the automatic labeling strategy;

[0315] The acquisition unit 306 is further configured to acquire the number of the automatic labeling strategies in the automatic labeling strategy set;

[0316] The above-mentioned determination unit 305 is further used to determine whether the above-mentioned number of strategies is less than or equal to a strategy number threshold;

[0317] If it is less than, the processing unit 308 is further used to traverse the annotation quality set and the cost saving set, and the calculation unit 307 is further used to calculate the Pareto dominance relationship between any two of the automatic annotation strategies;

[0318] In some optional implementations, the determining unit 302 is further configured to determine a first automatic labeling strategy based on the Pareto dominance relationship, including:

[0319] The calculation unit 307 is further used to calculate the second number of remaining automatic labeling strategies dominated by each automatic labeling strategy according to the Pareto dominance relationship, where the remaining automatic labeling strategies refer to the remaining automatic labeling strategies after any automatic labeling strategy is selected from the automatic labeling strategy set;

[0320] The processing unit 308 is further configured to prioritize the automatic labeling strategies in the automatic labeling strategy set according to the second number, and the determining unit 302 is further configured to determine the automatic labeling strategy with the highest second number as the first automatic labeling strategy;

[0321] If it is not less than, the processing unit 308 is further configured to use the automatic labeling strategy set as a decision space of a genetic algorithm to obtain the first automatic labeling strategy through the genetic algorithm.

[0322] In some optional implementations, the judgment unit 305 is further configured to judge whether the first automatic labeling strategy satisfies a first quality assessment indicator, including:

[0323] The processing unit 308 is further configured to label the quality inspection data using the first automatic labeling strategy;

[0324] The judgment unit 305 is further configured to judge whether the first automatic annotation strategy is greater than a quality control threshold of a first quality assessment indicator according to the annotation result of the quality inspection data by the first automatic annotation strategy and the manual annotation result of the quality inspection data;

[0325] If yes, the determination unit 302 is further configured to determine that the first automatic labeling strategy satisfies a first quality assessment indicator;

[0326] If not, the determining unit 302 is further configured to determine that the first automatic labeling strategy does not meet the first quality assessment indicator.

[0327] In some optional implementations, the determining unit 302 is further configured to determine that the first automatic labeling strategy does not meet the first quality assessment indicator, and further includes:

[0328] The determination unit 302 is further configured to determine the automatically labeled data of the target labeling task as unlabeled data;

[0329] The processing unit 308 is further configured to perform the following sampling and labeling operations on the unlabeled data to update the first automatic labeling strategy until the Pareto objective value of the first automatic labeling strategy converges:

[0330] The processing unit 308 is further configured to perform random sampling in the unlabeled data to obtain random sampling data;

[0331] The calculation unit 307 is further used to calculate the contribution value of each data item in the random sampling data to the multi-objective optimization model, and the multi-objective optimization model is used to calculate the Pareto optimal target value of the first automatic labeling strategy;

[0332] The processing unit 308 is further configured to select data items from the random sampling data to form high contribution value sampling data according to the contribution value of each data item in the random sampling data;

[0333] The above-mentioned labeling unit 304 is also used to manually label the above-mentioned high contribution value sampling data to obtain corresponding manual labeling results;

[0334] The calculation unit 307 is further used to recalculate the Pareto optimal target value of the first automatic labeling strategy based on the high contribution value sampling data and the corresponding manual labeling results. The processing unit 308 is further used to update the first automatic labeling strategy based on the high contribution value sampling data and the corresponding manual labeling results, and to update the unlabeled data based on the high contribution value sampling data and the corresponding manual labeling results, and to continue to perform the sampling and labeling operations on the updated unlabeled data until the Pareto target value converges.

[0335] It should be noted that the implementation details and technical effects of each unit in the data labeling device provided in the embodiments of the present disclosure can be referred to the description of other embodiments in the present disclosure, and will not be repeated here.

[0336] Reference below Figure 4 , which shows a schematic diagram of the structure of a computer system 400 of a terminal device suitable for implementing the present disclosure. Figure 4 The computer system 400 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0337] like Figure 4 As shown, the computer system 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 to a random access memory (RAM) 403. Various programs and data required for the operation of the computer system 400 are also stored in the RAM 403. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 404 is also connected to the bus 404.

[0338] Typically, the following devices may be connected to the I / O interface 404: input devices 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, etc.; output devices 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 408 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 409. The communication devices 409 may allow the computer system 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The computer system 400 of an electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0339] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 409, or installed from a storage device 408, or installed from a ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0340] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0341] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0342] The computer readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device implements the following Figure 2 The illustrated embodiment and its optional implementations illustrate a data labeling method.

[0343] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0344] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0345] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, a receiving unit may also be described as a "unit for receiving a data annotation request".

[0346] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

Claims

1. A data labeling method, characterized in that: The method comprises: Receive a data annotation request, wherein the data annotation request includes at least one annotation task and a task type, annotation data, constraints, and annotation index values ​​corresponding to each annotation task, wherein the annotation data includes at least one data item, and the constraints include at least one constraint condition; Determine a target labeling task according to the labeling indicator value of each labeling task; dividing the labeled data of the target labeling task into automatically labeled data and manually labeled data according to the constraint; Determine a target automatic labeling strategy according to at least one of a task type, labeling quality, and cost saving of the task to be labeled; The automatically labeled data is labeled using the target automatic labeling strategy.

2. The method according to claim 1, characterized in that The step of determining the target automatic labeling strategy according to at least one of the task type, labeling quality and cost saving of the task to be labeled includes: Determining whether there is a historical automatic labeling strategy corresponding to the target labeling task according to the task type of the target labeling task; If so, determining the historical automatic labeling strategy as the target historical automatic labeling strategy, and determining whether the target historical automatic labeling strategy meets the first quality assessment indicator; If not satisfied, acquiring an automatic labeling strategy set according to the task type of the target labeling task, the automatic labeling strategy set including at least one automatic labeling strategy; Based on at least one automatic labeling strategy, a target automatic labeling strategy is determined.

3. The method according to claim 2, characterized in that The step of determining the target automatic labeling strategy based on at least one automatic labeling strategy includes: Calculate the annotation quality and cost savings of each automatic annotation strategy; Determining a first automatic labeling strategy according to the labeling quality and the cost saving; Determining whether the first automatic labeling strategy meets a first quality assessment indicator; If so, the first automatic labeling strategy is determined to be the target automatic labeling strategy.

4. The method according to claim 1, characterized in that: The determining a target labeling task according to the labeling indicator value of each labeling task includes: Calculating, according to the labeling indicator value of each labeling task, a first number of remaining labeling tasks dominated by each labeling task, wherein the remaining labeling tasks refer to the labeling tasks remaining after any labeling task is selected from the at least one labeling task; Prioritize the at least one labeling task according to the first number, and determine the labeling task with the highest first number as the target labeling task.

5. The method according to claim 1, characterized in that The step of dividing the labeled data of the target labeling task into automatically labeled data and manually labeled data according to the constraint includes: Determine whether each data item in the labeled data satisfies each constraint condition of the constraint corresponding to the target labeling task; If satisfied, determining the data item of the annotated data as automatically annotated data; If not satisfied, the data item of the labeled data is determined as manually labeled data.

6. The method according to claim 5, characterized in that The step of determining the data item of the labeled data as manually labeled data further includes: Determining a target pre-filtering strategy according to the task type of the target labeling task; According to the target pre-filtering strategy, each data item in the manually labeled data is judged to determine whether it meets the target pre-filtering strategy threshold; If satisfied, the data item is sent to a manually labeled data item queue; If not satisfied, the data item is determined as a negative example and discarded.

7. The method according to claim 6, characterized in that The determining of the target pre-filtering strategy according to the task type of the target labeling task includes: Acquire a pre-filtering strategy set, the pre-filtering strategy set including at least one pre-filtering strategy, each of the pre-filtering strategies corresponding to a task type of a filterable annotation task, and the pre-filtering strategy is used to discard negative examples in the manually annotated data; Matching the task type of the target marking task with the task type of the filterable marking task corresponding to the pre-filtering strategy, and determining whether the task type of the target marking task exists in the task type of the filterable marking task corresponding to the pre-filtering strategy; If so, the pre-filtering strategy corresponding to the task type of the target labeling task is determined as the target pre-filtering strategy.

8. The method according to claim 2, characterized in that: The determining, according to the task type of the target labeling task, whether there is a historical automatic labeling strategy corresponding to the target labeling task includes: Acquire a historical automatic annotation strategy set, wherein the historical automatic annotation strategy set includes at least one historical automatic annotation strategy, each of which corresponds to a task type of an annotable task; and the historical automatic annotation strategy is used to annotate the automatically annotated data; Matching the task type of the target labeling task with the task type of the labelable task corresponding to the historical automatic labeling strategy; It is determined whether there is the historical automatic labeling strategy corresponding to the target labeling task according to whether there is the task type of the target labeling task among the task types of labelable tasks corresponding to the historical automatic labeling strategy.

9. The method according to claim 8, characterized in that The determining whether there is the historical automatic labeling strategy corresponding to the target labeling task according to whether there is the task type of the target labeling task among the task types of the labelable tasks corresponding to the historical automatic labeling strategy also includes: If not present, the automatically labeled data is determined as manually labeled data.

10. The method according to claim 2, characterized in that The determining whether the target history automatic labeling strategy meets the first quality assessment indicator includes: Randomly extracting data with a preset quality inspection ratio from the automatically labeled data, and determining the data with the preset quality inspection ratio as quality inspection data; Annotating the quality inspection data using the target history automatic annotation strategy; Obtaining the labeling results of the target history automatic labeling strategy on the quality inspection data and the manual labeling results on the quality inspection data; According to the labeling result of the quality inspection data by the target history automatic labeling strategy and the manual labeling result of the quality inspection data, it is judged whether the target history automatic labeling strategy meets the first quality assessment indicator.

11. The method according to claim 10, characterized in that The first quality assessment indicators include recall rate, precision rate, accuracy rate, false positive rate and false negative rate; and The determining whether the target history automatic labeling strategy meets the first quality assessment indicator according to the labeling result of the quality inspection data by the target history automatic labeling strategy and the manual labeling result of the quality inspection data includes: Calculating the recall rate, the precision rate, the accuracy rate, the false positive rate, and the false negative rate based on the labeling results of the quality inspection data by the target history automatic labeling strategy and the manual labeling results of the quality inspection data; Determining whether the target history automatic labeling strategy meets the first quality assessment indicator according to whether the recall rate, the precision rate, the accuracy rate, the false positive rate, and the false negative rate are respectively greater than a recall rate threshold, a precision rate threshold, an accuracy rate threshold, a false positive rate threshold, and a false negative rate threshold; If the recall rate, the precision rate, the accuracy rate, the false positive rate, and the false negative rate are respectively greater than the recall rate threshold, the precision rate threshold, the accuracy rate threshold, the false positive rate threshold, and the false negative rate threshold, it is determined that the target history automatic labeling strategy meets the first quality assessment indicator; If at least one of the recall rate, the precision rate, the accuracy rate, the false positive rate and the false negative rate is not greater than the recall rate threshold, the precision rate threshold, the accuracy rate threshold, the false positive rate threshold and the false negative rate threshold, it is determined that the target history automatic labeling strategy does not meet the first quality evaluation indicator.

12. The method according to claim 2, characterized in that: The determining whether the target history automatic labeling strategy meets the first quality assessment indicator also includes: If satisfied, the automatically labeled data is labeled using the target history automatic labeling strategy.

13. The method according to claim 2, characterized in that The step of acquiring an automatic labeling strategy set according to the task type of the target labeling task includes: Selecting X feature extraction models and Y classifier models from a predefined feature extraction model library and a predefined classifier model library according to the task type of the target labeling task, respectively, where X and Y are both positive integers; Any feature extraction model among the X feature extraction models and any classifier model among the Y classifier models are arranged and combined to obtain an automatic labeling strategy set.

14. The method according to claim 3, characterized in that The calculation of the annotation quality and cost saving of each automatic annotation strategy includes: Using each of the automatic labeling strategies to label the quality inspection data respectively; Calculating the labeling quality of each automatic labeling strategy according to the labeling result of each automatic labeling strategy on the quality inspection data and the manual labeling result of the quality inspection data; Acquiring task characteristics of the target labeling task, wherein the task characteristics include the number of data items of the automatically labeled data and the average cost of manually labeling a single data item of the automatically labeled data; The cost saving of each automatic labeling strategy is calculated according to the task characteristics and the labeling results of the quality inspection data by each automatic labeling strategy.

15. The method according to claim 14, characterized in that The calculating the labeling quality of each automatic labeling strategy according to the labeling result of each automatic labeling strategy on the quality inspection data and the manual labeling result of the quality inspection data includes: The recall rate, the precision rate, the accuracy rate and the proportion of correctly identified negative examples of each automatic labeling strategy are calculated respectively according to the labeling results of the quality inspection data by each automatic labeling strategy and the manual labeling results of the quality inspection data. The recall rate, the precision rate, the accuracy rate and the proportion of correctly identified negative examples are linearly weighted to obtain the labeling quality of each automatic labeling strategy.

16. The method according to claim 3, characterized in that The determining of the first automatic labeling strategy according to the labeling quality and the cost saving includes: Based on the annotation quality and the cost saving, calculating the Pareto dominance relationship between any two of the automatic annotation strategies; Based on the Pareto dominance relationship, a first automatic labeling strategy is determined.

17. The method according to claim 16, characterized in that The calculating the Pareto dominance relationship between any two of the automatic labeling strategies based on the labeling quality and the cost saving includes: Establishing a labeling quality set and a cost saving set of the automatic labeling strategy; Obtaining the number of strategies of the automatic marking strategy in the automatic marking strategy set; Determine whether the number of strategies is less than or equal to a strategy number threshold; If it is less than, traverse the annotation quality set and the cost saving set to calculate the Pareto dominance relationship between any two of the automatic annotation strategies.

18. The method according to claim 16 or 17, characterized in that The determining of a first automatic labeling strategy based on the Pareto dominance relationship includes: Calculate, according to the Pareto dominance relationship, a second number of remaining automatic labeling strategies dominated by each automatic labeling strategy, where the remaining automatic labeling strategies refer to the automatic labeling strategies remaining after any one of the automatic labeling strategies is selected from the automatic labeling strategy set; Prioritize the automatic labeling strategies in the automatic labeling strategy set according to the second number, and determine the automatic labeling strategy with the highest second number as the first automatic labeling strategy; If it is not less than, the automatic labeling strategy set is used as the decision space of the genetic algorithm, and the first automatic labeling strategy is obtained by the genetic algorithm.

19. The method according to claim 3, characterized in that The determining whether the first automatic labeling strategy meets the first quality assessment indicator includes: Using the first automatic labeling strategy to label the quality inspection data; Determining whether the first automatic labeling strategy is greater than a quality control threshold of a first quality assessment indicator according to a labeling result of the quality inspection data by the first automatic labeling strategy and a manual labeling result of the quality inspection data; If so, determining that the first automatic labeling strategy satisfies a first quality assessment indicator; If not, it is determined that the first automatic labeling strategy does not meet the first quality assessment indicator.

20. The method according to claim 19, characterized in that The determining that the first automatic labeling strategy does not meet the first quality assessment indicator also includes: Determining the automatically labeled data of the target labeling task as unlabeled data; The following sampling and labeling operations are performed on the unlabeled data to update the first automatic labeling strategy until the Pareto objective value of the first automatic labeling strategy converges: Performing random sampling in the unlabeled data to obtain random sampling data; Calculating the contribution value of each data item in the randomly sampled data to a multi-objective optimization model, wherein the multi-objective optimization model is used to calculate the Pareto optimal target value of the first automatic labeling strategy; According to the contribution value of each data item in the random sampling data, data items are selected from the random sampling data to form high contribution value sampling data; Manually labeling the high contribution value sampling data to obtain corresponding manual labeling results; According to the high contribution value sampling data and the corresponding manual labeling results, the Pareto optimal target value of the first automatic labeling strategy is recalculated, the first automatic labeling strategy is updated based on the high contribution value sampling data and the corresponding manual labeling results, and the unlabeled data is updated according to the high contribution value sampling data and the corresponding manual labeling results, and the sampling and labeling operations are continued to be performed on the updated unlabeled data until the Pareto target value converges.

21. A data annotation device, comprising: A receiving unit, configured to receive a data annotation request, wherein the data annotation request includes at least one annotation task and a task type, annotation data, constraints, and annotation index values ​​corresponding to each annotation task, wherein the annotation data includes at least one data item, and the constraints include at least one constraint condition; A determination unit, configured to determine a target labeling task according to the labeling indicator value of each labeling task; a segmentation unit, which segments the labeled data of the target labeling task into automatically labeled data and manually labeled data according to the constraint; The determination unit is further used to determine the target automatic labeling strategy according to at least one of the task type, labeling quality and cost saving of the task to be labeled; A labeling unit is used to label the automatic labeling data using the target automatic labeling strategy.

22. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method according to any one of claims 1 to 20.

23. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by one or more processors, the method according to any one of claims 1 to 20 is implemented.

Images