Data annotation method and device, equipment and storage medium

Abstract

The invention provides a data annotation method and device, equipment and a storage medium, and relates to the technical field of data annotation, and the method comprises the steps: obtaining a plurality of candidate tags of to-be-annotated data; determining a label fitting scheme according to the label types to which the plurality of candidate labels belong and the data features of the to-be-labeled data; the label type comprises a classification label, a detection box label and a text label; one label type corresponds to one data fitting scheme; and according to a label fitting scheme, fitting the plurality of candidate labels, and determining a target label of the to-be-labeled data. On the basis, the accuracy of a data labeling result can be improved.

Description

Technical Field

[0001] This application relates to the field of data annotation technology, and in particular to a data annotation method, apparatus, device, and storage medium. Background Technology

[0002] Data labeling is a core step in building supervised learning datasets. Its goal is to assign accurate semantic labels to raw data (such as images and text), thereby providing machine learning models with learning samples that have clear supervisory signals.

[0003] In existing data annotation techniques, a multi-annotation fitting method is often used to improve the reliability of annotation results. Specifically, after generating multiple annotation results for the data to be annotated, the candidate annotation results are fitted to determine the final annotation result.

[0004] However, existing data annotation technologies may struggle to achieve accurate fitting of annotation results when faced with data annotation scenarios that vary greatly in complexity, thus reducing the accuracy of data annotation. Summary of the Invention

[0005] This application provides a data annotation method, apparatus, device, and storage medium that can improve the accuracy of data annotation.

[0006] To address the above problems, the embodiments of this application provide the following technical solutions:

[0007] Firstly, this application provides a data annotation method, which includes: obtaining multiple candidate labels for data to be annotated; determining a label fitting scheme based on the label types of the multiple candidate labels and the data characteristics of the data to be annotated; the label types include classification labels, bounding box labels, and text labels; one label type corresponds to one data fitting scheme; and fitting the multiple candidate labels according to the label fitting scheme to determine the target label for the data to be annotated.

[0008] The data annotation method provided in this application first determines a label fitting scheme by considering the label types of multiple candidate labels and the data characteristics of the data to be annotated. The label types of the candidate labels clarify the differentiated fitting rules for different label types. The data characteristics of the data to be annotated allow analysis of its complexity (such as the depth of text nesting in text data), and based on this, the fitting threshold for candidate labels is dynamically adjusted, enabling the candidate label selection process to be optimized in conjunction with the data characteristics of the data to be annotated. Finally, by using the fitting scheme adjusted by the data characteristics to fit multiple candidate labels, the fitting efficiency can be improved, and more accurate target labels can be obtained. Compared with existing technologies, this application overcomes the limitations of traditional fixed fitting threshold determination. Through the collaborative analysis of data characteristics and label types, it realizes a data fitting method for data to be annotated with different data characteristics, effectively improving the annotation accuracy under different complexity scenarios.

[0009] One possible implementation involves multiple candidate labels of category type. The data characteristics of the data to be labeled include the number of categories for each candidate label. Based on a label fitting scheme, the multiple candidate labels are fitted to determine the target label for the data to be labeled. This includes selecting the candidate label with the highest label score from the multiple candidate labels as the first candidate label. The label score refers to the sum of the labeling accuracy rates of all annotators who labeled the same candidate label. The ratio of the label score of the first candidate label to the sum of the labeling accuracy rates of all annotators is used as the confidence level of the first candidate label. If the first candidate label is a single label and its confidence level is greater than a confidence threshold, the first candidate label is determined as the target label. The confidence threshold is the larger of a first threshold and a second threshold. The first threshold is a preset fixed value, and the second threshold is determined based on the number of categories for each candidate label and the average label score of the multiple candidate labels.

[0010] One possible implementation involves multiple candidate labels, each with a detection box label type. The data features of the data to be labeled include the image area and the number of targets to be detected. Based on a label fitting scheme, multiple candidate labels are fitted to determine the target label for the data to be labeled. This includes selecting the candidate label with the largest average detection box intersection-union ratio (IU / U) that is greater than a third threshold as the second candidate label. The average IU / U of a candidate label refers to the average IU / U of the candidate label with each other. The third threshold is determined based on the image area and the number of targets to be detected in the data to be labeled. If the second candidate label is a single label and its average IU / U meets preset conditions, the second candidate label is determined as the target label. These preset conditions include: the ratio of the average IU / U of the second candidate label to the average IU / U of the third candidate label is greater than a fourth threshold. The third candidate label is the label whose average IU / U is the second smallest than that of the second candidate label.

[0011] One possible implementation involves multiple candidate labels of text type. The data characteristics of the data to be labeled include the nesting depth of the text entities. Based on a label fitting scheme, multiple candidate labels are fitted to determine the target label for the data to be labeled. This includes: selecting the candidate label with the lowest edit score from among the multiple candidate labels as the fourth candidate label; the edit score of a candidate label refers to the sum of the edit distances between that candidate label and other candidate labels; the edit distance refers to the number of times a candidate label is changed to another candidate label; if the fourth candidate label is a single label and its label score is less than a fifth threshold, the fourth candidate label is determined as the target label; the fifth threshold is determined based on the nesting depth of the text entities in the data to be labeled.

[0012] One possible implementation involves conflict arbitration of the target candidate label under preset arbitration conditions. The target candidate label can be any one of the first candidate label, the second candidate label, and the third candidate label.

[0013] One possible implementation involves conflict arbitration of target candidate labels, including: obtaining arbitration labels for the data to be labeled. These arbitration labels are obtained by arbitrators annotating the data. For first candidate labels, the text similarity between each first candidate label and the arbitration label is calculated. If the text similarity is greater than a similarity threshold and the confidence level of the arbitration label is greater than a first arbitration threshold, the arbitration label is determined as the target label for the data to be labeled. For second candidate labels, the maximum offset distance between each second candidate label and the arbitration label is calculated. If the maximum offset distance is less than an offset threshold and the confidence level of the arbitration label is greater than a second arbitration threshold, the arbitration label is determined as the target label for the data to be labeled. For third candidate labels, the minimum edit distance between each third candidate label and the arbitration label is calculated. If the minimum edit distance is less than an edit distance threshold and the confidence level of the arbitration label is greater than a third arbitration threshold, the arbitration label is determined as the target label for the data to be labeled. If the arbitration label is not determined as the target label for the data to be labeled, manual arbitration is performed on the data to be labeled.

[0014] One possible implementation involves, before obtaining multiple candidate labels for the data to be labeled, the method further including: determining the corresponding labeling model based on the data type of the data to be labeled; inputting the data to be labeled into the labeling model to obtain the labeling results and corresponding confidence levels; determining the labeling mode based on the confidence levels corresponding to the labeling results; and using the labeling mode to obtain multiple candidate labels for the data to be labeled.

[0015] Secondly, this application provides a data annotation apparatus, which includes various functional modules for the method described in the first aspect above.

[0016] Thirdly, this application provides an electronic device comprising a processor and a memory. The memory stores processor-executable instructions, which, when configured to execute the instructions, cause the electronic device to perform the method described in the first aspect above.

[0017] Fourthly, this application provides a readable storage medium comprising software instructions. When the software instructions are executed in an electronic device, they cause the electronic device to perform the method described in the first aspect above.

[0018] Fifthly, this application provides a computer program product comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method described in the first aspect.

[0019] The beneficial effects of the second to fifth aspects mentioned above can be referred to the first aspect, and will not be repeated here. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating a data annotation method provided in an embodiment of this application;

[0022] Figure 2 A flowchart illustrating a data annotation method for a classification type provided in an embodiment of this application;

[0023] Figure 3 A flowchart illustrating a data annotation method for a detection box type provided in an embodiment of this application;

[0024] Figure 4 A flowchart illustrating a text-type data annotation method provided in an embodiment of this application;

[0025] Figure 5 A flowchart illustrating a conflict arbitration method provided in an embodiment of this application;

[0026] Figure 6 A flowchart illustrating a candidate tag acquisition method provided in an embodiment of this application;

[0027] Figure 7 This is a schematic diagram illustrating the application process of a data annotation method provided in an embodiment of this application;

[0028] Figure 8 This is a schematic diagram illustrating the composition of a data annotation device provided in an embodiment of this application;

[0029] Figure 9 This is a schematic diagram of the composition of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] Hereinafter, the terms "first," "second," and "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first," "second," or "third," etc., may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0031] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0032] Existing data annotation techniques typically use fixed fitting rules to fit multiple annotation results, which ignores the data complexity of the labeled data itself.

[0033] For example, when the amount of information in the data to be labeled is relatively small, the direct result in the labeling results is that multiple candidate labels will show a high degree of similarity. Conversely, when the data to be labeled is rich and complex, the candidate labels may show significant differences.

[0034] In this case, a fixed fitting rule may have the following problems: applying an overly strict fitting standard to candidate labels with high similarity will lead to the loss of effective information, while applying a lenient fitting standard to candidate labels with large differences may introduce noise.

[0035] Therefore, how to consider the data characteristics of the data to be labeled during the data labeling process has become an urgent problem to be solved.

[0036] The data annotation method provided in this application can determine a label fitting scheme for multiple candidate labels of the data to be annotated based on the data characteristics of the data to be annotated. Specifically, based on the data characteristics, the fitting threshold in the label fitting scheme is adjusted. Different fitting thresholds are applied to the data to be annotated with different data complexities, which can improve the accuracy of data fitting and thus improve the accuracy of annotation results in multi-label data annotation scenarios.

[0037] The blockchain business data detection method provided in this application can be applied to computing devices, wherein the computing devices have the ability to communicate with network nodes in the network. Specifically, the computing device can be a single server or a server cluster consisting of multiple servers, or a computer, or a processor or processing chip in a server or computer, etc. This application does not limit the specific form of the computing device.

[0038] The data annotation method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0039] like Figure 1 As shown, when this method is applied to a computing device, it includes the following:

[0040] S101. Obtain multiple candidate labels for the data to be labeled.

[0041] One possible implementation is that multiple candidate labels can be obtained by multiple annotators independently labeling the data to be labeled, and then manually input into the computing device.

[0042] Another possible implementation is that multiple candidate labels can be obtained by labeling the data to be labeled using different labeling models. In this case, the computing device first receives the data to be labeled, and then determines multiple corresponding labeling models based on the labeling type of the data to be labeled (such as classification labeling, image detection labeling, etc.). Then, it uses the determined multiple labeling models to obtain the data to be labeled and the labeling type.

[0043] Another possible approach is to use a combination of annotation models and annotators to obtain candidate labels.

[0044] While manual annotation can incorporate human subjective understanding to make judgments that fit the actual scenario for complex data (such as text with ambiguous semantics or images with interwoven elements), significant differences in subjective understanding among annotators can lead to biased candidate labels. Using annotation models, while fast, with a uniform output format, and capable of quickly processing large-scale data, is less adaptable to subtle semantics or specific scenarios implicit in the annotated data, easily generating candidate labels that deviate from real-world needs. Therefore, computing devices can first use annotation models to obtain a pre-annotation result, limiting the annotation range to a reasonable interval. Then, annotators can verify and correct the pre-annotation result, resulting in candidate labels that are both consistent and accurate.

[0045] For details on the implementation methods described above, please refer to the following text. Figure 6 This will not be elaborated upon here.

[0046] S102. Determine the label fitting scheme based on the label types of multiple candidate labels and the data characteristics of the data to be labeled.

[0047] The label types include category labels, bounding box labels, and text labels. Each label type corresponds to a data fitting scheme.

[0048] Because multiple candidate labels are obtained by labeling the data according to the data labeling type of the data to be labeled, the label types of multiple candidate labels are consistent.

[0049] Category labels refer to labels obtained by classifying data based on its core category features, and used to define the category to which the data belongs.

[0050] For example, in image data, images are labeled as "cat," "tree," or "car" based on category features such as "animal," "plant," or "vehicle." In text data, text is labeled as "positive," "negative," or "neutral" based on the "sentiment" feature. These are all classification labels.

[0051] Detection box labels refer to labels used in visual data such as images or videos to accurately locate target objects and annotate their category information by defining boundaries such as rectangular boxes.

[0052] For example, in street view images, objects such as pedestrians and vehicles are enclosed in rectangular boxes and labeled with "pedestrian," "car," "truck," etc., to clearly identify the location and category of the target.

[0053] Text labels refer to tags generated for text-based data to be labeled by extracting semantic details, marking specific attributes, or supplementing explanatory information. They are usually presented in the form of natural language phrases or short sentences and are used to refine text features rather than simply classify them.

[0054] For example, the text "A certain brand's new mobile phone has a battery life of up to 48 hours" is labeled with a descriptive tag such as "strong battery life"; the text "The meeting will be held next Monday at 9 o'clock" is labeled with attribute tags such as "Time: Next Monday at 9 o'clock" and "Type: Meeting" - these tags focus more on breaking down or supplementing the details of the text content, rather than just classifying it into major categories.

[0055] By introducing the label types, it can be understood that the data formats of different label types are different, and the degree of difference and the form of expression between different candidate labels of different data to be labeled will also vary depending on the data characteristics of the data to be labeled.

[0056] For example, if the data to be labeled is a simple image (such as an image of a single object against a plain white background, like an apple against a white background), the differences in candidate labels are usually small, possibly only manifesting as subtle deviations in classification labels (such as "red apple" and "apple"), or slight offsets in the edge positions of bounding box labels. If the data to be labeled is a complex image (such as a street view containing multiple overlapping objects, including pedestrians, vehicles, billboards, etc.), the differences in candidate labels will be more obvious: classification labels may differ between "pedestrian" and "passerby," and bounding box labels may have significant discrepancies in the selection range of overlapping areas (like an obscured car, some bounding boxes include part of the background, while others only select the visible car body).

[0057] Therefore, if the fitting scheme is based solely on uniform rules without considering the differences in label types and data characteristics (such as data complexity and semantic clarity) between the candidate labels, the fitting results may not match the actual data requirements. Therefore, it is necessary to determine the label fitting scheme based on the label types and data characteristics of the data to be labeled.

[0058] Specifically, when fitting multiple candidate labels, the corresponding fitting rules are first determined according to the label types to which the multiple candidate labels belong (the fitting rules correspond to the data format). Then, the threshold of each fitting rule is adjusted according to the data characteristics of the data to be labeled, so as to obtain a label fitting scheme that fits multiple candidate labels.

[0059] S103. According to the label fitting scheme, fit multiple candidate labels to determine the target label of the data to be labeled.

[0060] Because multiple candidate labels may contain the same candidate label, it is necessary to remove duplicate candidate labels before fitting them.

[0061] For a detailed description of step S103, please refer to the following text. Figure 2 , Figure 3 , Figure 4 Detailed explanations are omitted here.

[0062] The data annotation method provided in this application first determines a label fitting scheme by considering the label types of multiple candidate labels and the data characteristics of the data to be annotated. The label types of the candidate labels clarify the differentiated fitting rules for different label types. The data characteristics of the data to be annotated allow analysis of its complexity (such as the depth of text nesting in text data), and based on this, the fitting threshold for candidate labels is dynamically adjusted, enabling the candidate label selection process to be optimized in conjunction with the data characteristics of the data to be annotated. Finally, by using the fitting scheme adjusted by the data characteristics to fit multiple candidate labels, the fitting efficiency can be improved, and more accurate target labels can be obtained. Compared with existing technologies, this application overcomes the limitations of traditional fixed fitting threshold determination. Through the collaborative analysis of data characteristics and label types, it realizes a data fitting method for data to be annotated with different data characteristics, effectively improving the annotation accuracy under different complexity scenarios.

[0063] The following section, in conjunction with the accompanying diagram, introduces the label fitting schemes corresponding to the classification labels.

[0064] In some embodiments, the label type of the multiple candidate labels can be classification labels, and the data features of the data to be labeled include the number of classification categories of the multiple candidate labels of the data to be labeled. In this case, the implementation process of step S103 is as follows: Figure 2 As shown, step S103 specifically includes the following steps:

[0065] S201. From multiple candidate labels, select the candidate label with the highest label score as the first candidate label.

[0066] The label score refers to the sum of the labeling accuracy rates of all labelers who labeled the same candidate label.

[0067] The label score reflects the degree to which a candidate label is recognized by the labelers. The higher the score, the more people with high labeling ability recognize the label, and the higher the probability that it matches the true characteristics of the data. Therefore, the candidate label with the highest label score is selected as the first candidate label.

[0068] For a candidate label of a category label type, it is necessary to know the annotation accuracy of all annotators who labeled that candidate label. Therefore, in the process of fitting candidate labels for category labels, it is not necessary to deduplicate multiple candidate labels.

[0069] For example, the calculation process for the label score of each candidate label can be represented as follows:

[0070]

[0071] Among them, Score i This refers to the label score of the i-th candidate label. n refers to the total number of labelers. j refers to the j-th labeler, δ(L j L i `(j)` is an indicator function used to determine whether the candidate label labeled by the j-th annotator is the same as the i-th candidate label. If they are the same, the indicator function returns 1; otherwise, it returns 0. j This refers to the annotation accuracy rate of the j-th annotator.

[0072] For example, suppose there are 3 annotators (n=3) with annotation accuracy rates of 0.9, 0.8, and 0.7 respectively. For the data to be annotated, there are two candidate labels, A and B. The first and second annotators labeled candidate label A, and the third annotator labeled candidate label B. Then, for candidate label A (i=A), δ(L1,A)=1, δ(L2,A)=1, and δ(L3,A)=0. According to formula (1), the score can be obtained. A=1×0.9+1×0.8+0×0.7=1.7. For candidate label B (i=B), δ(L1,B)=0, δ(L2,B)=0, δ(L3,B)=1, we can obtain the Score. B = 0 × 0.9 + 0 × 0.8 + 1 × 0.7 = 0.7, meaning candidate label A has a label score of 1.7 and candidate label B has a label score of 0.7. Therefore, candidate label A is the candidate label with the highest label score, which is the first candidate label.

[0073] In some embodiments, the calculation process for the annotation accuracy of annotation personnel can be expressed as follows:

[0074]

[0075] in, This refers to the annotation accuracy rate after the t-th data annotation by annotation personnel j. This refers to the annotation accuracy of annotator j after the (t-1)th data annotation, where α is the update decay factor, and N... total This refers to the number of times annotator j annotated, N correct This refers to the number of times that the candidate label labeled by labeler j is identified as the target label.

[0076] For example, the value of α can be 0.9, 0.8 or 0.7, and the specific value of α is not limited in the embodiments of this application.

[0077] For example, suppose that the annotation accuracy of annotator j after the (t-1)th annotation is 0.85 and the update decay factor is 0.7. After the tth data annotation, the annotator has annotated the data 91 times, of which 77 candidate labels are finally determined as target labels. Then, according to formula (2), the accuracy after the tth annotation can be calculated as: 0.7×0.85+(1-0.7)×(91 / 77)=0.849.

[0078] S202. The ratio of the label score of the first candidate label to the sum of the labeling accuracy of all labelers is used as the confidence level of the first candidate label.

[0079] When calculating the confidence level of the first candidate label, the ratio of its label score to the sum of the labeling accuracy rates of all annotators is used as the indicator. This method objectively reflects the relative acceptance of the first candidate label among all labeling results. The label score represents the total accuracy rate of the annotators who accept the candidate label, while the sum of the accuracy rates of all annotators is a quantitative representation of the overall labeling capability. If the average accuracy rate of all annotators is high, even if the absolute value of a candidate label's score is large, its proportion in the overall labeling capability may not be prominent. Conversely, if the ratio is high, it indicates that the candidate label has received a higher proportion of "effective acceptance" from the overall labeling resources, thus more accurately reflecting its reliability.

[0080] For example, suppose there are 3 annotators with annotation accuracies of 0.9, 0.8, and 0.7 respectively. The sum of the accuracies of all annotators is 0.9 + 0.8 + 0.7 = 2.4. If the first candidate label is annotated by the first two annotators, its label score is 0.9 + 0.8 = 1.7. Then the confidence level of the first candidate label is 1.7 ÷ 2.4 ≈ 0.71.

[0081] S203. If the first candidate label is a single label and the confidence level of the first candidate label is greater than the confidence level threshold, then the first candidate label is determined as the target label.

[0082] The confidence threshold is the larger of the first threshold and the second threshold. The first threshold is a preset fixed value, and the second threshold is determined based on the number of classification categories of multiple candidate labels and the average label score of multiple candidate labels.

[0083] The confidence threshold can also be understood as the fitting threshold, which is used to screen the effectiveness of the first candidate label.

[0084] The second threshold, calculated by combining the number of categories among multiple candidate labels with the average label score, directly reflects the labeling difficulty and overall consensus of the data to be labeled, transforming this characteristic into a quantifiable criterion. Specifically, the more categories there are, the smaller the second threshold becomes.

[0085] Setting the confidence threshold to the maximum of the first threshold (a preset fixed value) and the second threshold (dynamically calculated based on candidate label features) is to achieve a dual constraint: the first threshold serves as a baseline standard, ensuring that the target label at least reaches a preset basic reliability level, avoiding a decrease in annotation quality due to scenario-specificities. The second threshold is dynamically adjusted based on the number of candidate label categories (more categories indicate potentially more complex data, requiring higher consensus) and the average label score (reflecting the overall concentration of annotations), allowing the threshold to adapt to the data complexity of the data to be annotated.

[0086] In some embodiments, if the confidence level of the first candidate label is greater than or equal to the confidence level threshold, the first candidate label can also be determined as the target label.

[0087] For example, the process can be represented as:

[0088]

[0089] in, This refers to the tag score of the first candidate tag. This refers to the sum of the annotation accuracy rates of all annotators corresponding to multiple candidate labels. a, b, and c are constants, β refers to the number of classification categories of multiple candidate labels, and avg refers to the average label score of multiple candidate labels.

[0090] For example, the value of 'a' can be 0.7, 0.75, or 0.8; the value of 'b' can be 1.2, 1.3, or 104; and the value of 'c' can be 0.2, 0.3, or 0.4. Specifically, the values ​​of 'a', 'b', and 'c' can be adjusted according to the actual application scenario, and this application embodiment does not impose any restrictions on this.

[0091] For example, suppose the first candidate label has a label score of 4.5, the sum of the accuracy of all annotators is 5.0, and its confidence level is 4.5 ÷ 5.0 = 0.9; the first threshold is 0.75, the constants b and c are 1.2 and 0.3, the number of categories for multiple candidate labels is 2, and the average label score is 0.6. First, calculate the second threshold: (1.2 - 0.3 × 2) × 0.6 = 0.36 = 2.4. The confidence threshold is then max(0.75, 0.36) = 0.75. Since the confidence level (0.9) of the first candidate label is greater than the confidence threshold (0.75), this first candidate label is determined to be the target label.

[0092] In addition, if there are multiple first candidate labels, it means that there are multiple candidate labels with the same confidence level and recognized by most annotators. In this case, the fitting algorithm alone cannot select the first candidate label, and arbitration is required.

[0093] As shown in steps S201-S203, calculating the label score for each candidate label based on the labeling accuracy of the labelers allows the label scores to reflect the differences in labeling capabilities among the labelers, giving greater weight to the opinions of labelers with high accuracy rates. Simultaneously, calculating the confidence level of the first candidate label by summing the labeling accuracy rates of all labelers eliminates the influence of external factors such as the number of labelers, objectively reflecting the relative acceptance of the first candidate label among all labeling results. Furthermore, determining the confidence threshold based on data characteristics such as the number of classification categories allows the judgment criteria to be dynamically adjusted according to the complexity of the data to be labeled during the screening process, making the screening process closer to the actual situation of the data to be labeled. This improves the accuracy of the screening and yields more accurate target labels.

[0094] The following section, in conjunction with the accompanying diagram, introduces the label fitting scheme corresponding to the detection box label.

[0095] In some embodiments, the label type of the multiple candidate labels can be detection box labels, and the data features of the data to be labeled include the image area of ​​the data to be labeled and the number of targets to be detected in the data to be labeled. In this case, the implementation process of step S103 is as follows: Figure 3 As shown, step S103 specifically includes the following steps:

[0096] S301. From multiple candidate labels, select the candidate label with the largest average crossover ratio of the detection boxes that is greater than the third threshold as the second candidate label.

[0097] Here, the average value refers to the average of the intersection-union ratios (IoU) of a candidate label with every other candidate label. The third threshold is determined based on the image area and the number of targets to be detected in the data to be labeled.

[0098] The intersection-union ratio (IUU) is a quantitative indicator of the degree of overlap between two bounding boxes. It is calculated as the ratio of the intersection area to the union area of ​​the two bounding boxes (the value ranges from 0 to 1, and the closer the value is to 1, the higher the degree of overlap).

[0099] The average intersection-union ratio (IUR) of a candidate label's detection boxes refers to the arithmetic mean of the IURs of a candidate label's detection boxes with the detection boxes of all other candidate labels. Specifically, the IUR of a candidate label with each other is first obtained, then these IURs are summed and divided by the total number of other candidate labels to obtain the average IUR of a candidate label's detection boxes.

[0100] The average intersection-union ratio (IU) of the detection boxes can measure the consensus of the location of the detection boxes of a candidate label among all the annotation results. The higher the average IU of the detection boxes of the candidate label, the higher the overlap between the detection boxes of the candidate label and the majority of other detection boxes. In other words, the more consistent the annotators are in judging the location of the detection boxes in the annotation data. The lower the average, the more significant the difference between the location annotation of the detection box and other annotations, and the lower the consensus.

[0101] Meanwhile, the significance of comparing the third threshold with the average intersection-union ratio of the detection boxes lies in filtering out candidate labels that meet the requirements of the consensus on location labeling by setting a reasonable threshold, namely, the second candidate labels.

[0102] The third threshold is determined based on the image area and the number of targets to be detected in the data to be labeled. This allows the screening criteria to be adapted to the actual characteristics of the data. For example, in scenes with a large image area, the target may occupy a smaller proportion of the image, and even slight deviations in the detection boxes can lead to a decrease in the intersection-union ratio. In this case, the threshold needs to be appropriately lowered to avoid over-screening. This threshold calculation method can reflect the spatial complexity (image area) and target density (number of targets) of the data to be labeled, ensuring that the screening process neither misses reasonable annotations due to overly strict standards nor retains results with significant discrepancies due to overly lenient standards.

[0103] For example, the calculation process of the third threshold can be expressed as follows:

[0104]

[0105] Where θ3 represents the third threshold, θ 基础 This refers to the base threshold coefficient, N. target S refers to the total number of targets that need to be labeled in the image (data) to be labeled. 待标注数据 N refers to the area of ​​the image (data) to be labeled. target This refers to the number of targets to be detected in the image, where d and e are constants.

[0106] For example, θ 基础 d can be set to 0.8, 0.85, or 0.9. c can be set to 0.8, 0.9, or 0.95. The specific values ​​of c are not limited in the embodiments of this application.

[0107] S302. If the second candidate label is a single label and the average crossover ratio of the detection boxes of the second candidate label meets the preset conditions, the second candidate label is determined as the target label.

[0108] The preset conditions include: the ratio of the average intersection-union ratio of the detection boxes of the second candidate label to the average intersection-union ratio of the detection boxes of the third candidate label is greater than a fourth threshold; the third candidate label is the label whose average intersection-union ratio of the detection boxes is less than that of the second candidate label.

[0109] The second candidate label is a single label, ensuring the uniqueness of the target label. The ratio of the average intersection-union ratio (IU) of the second candidate label's largest detection bounding boxes to the average IU of the third candidate label's detection bounding boxes is greater than a fourth threshold, confirming that the second candidate label has a significant advantage. If the values ​​of the second and third candidate labels are close, it indicates that the consensus on the location annotation of different candidate labels is not significantly different, and there may be annotation discrepancies. However, when the ratio of their average IU of detection bounding boxes exceeds the fourth threshold, it indicates that the second candidate label has a significantly higher consensus on location annotation than the second largest one. This effectively avoids the difficulty in choosing due to multiple candidate labels performing similarly, while highlighting the reliability of the optimal candidate label, providing a clear basis for determining a unique target label, and reducing annotation errors caused by fuzzy judgments.

[0110] For example, the fourth threshold can be 1.3, 1.5 or 1.7, and the specific value of the fourth threshold is not limited in the embodiments of this application.

[0111] As shown in steps S301-S302, the second candidate label is first selected by using the average intersection-union ratio (IU / R) of the detection boxes. This second candidate label has the largest IU / R and is greater than a third threshold. The third threshold is dynamically adjusted based on the image area and the number of targets to adapt the selection to the data characteristics and ensure the second candidate label has basic reliability in location labeling. Then, by using the condition that the ratio of the IU / R of the second candidate label to the average IU / R of the third candidate label is greater than a fourth threshold, the significant advantage of the second candidate label is confirmed. This reduces decision ambiguity caused by multiple candidate labels performing similarly, ultimately accurately determining a unique and reliable target label. This ensures a consensus basis for location labeling while improving the certainty of the result through differentiated judgment.

[0112] The following section, in conjunction with the accompanying diagram, introduces the label fitting scheme for the text labels.

[0113] In some embodiments, the label type of the multiple candidate labels can be text labels, and the data features of the data to be labeled include the nesting depth of the text entities in the data to be labeled. In this case, the implementation process of step S103 is as follows: Figure 4 As shown, step S103 specifically includes the following steps:

[0114] S401. From multiple candidate tags, select the candidate tag with the lowest editing score as the fourth candidate tag.

[0115] In this context, the edit score of a candidate tag is the sum of its edit distances to all other candidate tags. Edit distance refers to the number of times a candidate tag is changed to another candidate tag.

[0116] Edit distance refers to the number of editing operations (such as inserting, deleting, and replacing characters) required to transform one text string into another. It quantifies the similarity between two texts; the smaller the value, the more similar the texts are. Therefore, the candidate tag with the lowest edit distance score is most similar to the other candidate tags.

[0117] In some embodiments, because the original value of the edit distance is affected by the text length—for example, the number of edit operations between long texts is naturally greater than that between short texts (even if the similarity ratio is the same), directly using it for calculation may lead to misjudgment of the similarity of long texts. Therefore, after the computing device calculates the edit distance between each pair of annotation results, it can normalize the edit distance (e.g., divide it by the maximum or sum of the lengths of the two texts) to eliminate the influence of the candidate label text length and make the similarity comparison of texts of different lengths more objective.

[0118] For example, the normalization calculation process can be represented as follows:

[0119]

[0120] NormDistr i，j This refers to the formatted edit distance between candidate label i and another candidate label j, EditDist(t i ,t j ) refers to the initial edit distance between candidate label i and another candidate label j, len(t) i ) refers to the text length of candidate label i, len(t) j ) refers to the text length of candidate label j.

[0121] In some embodiments, to more accurately measure text similarity, especially when dealing with labeled data containing complex nested entity depths, the text nested entity depth of the labeled data can be used to enhance it.

[0122] The text entity nesting depth of the data to be labeled refers to the number of levels of entity inclusion relationships in the text. For example, in "XX Company (headquartered in City A) Department B", "XX Company" contains "Department B", and the nesting depth can be considered as 2, which reflects the complexity of the text entity structure.

[0123] For example, the process of using text nesting depth to enhance the third tag for each formatted edit distance can be represented as:

[0124] Disti，j =NormDistr i，j ×(1+f×depth 待标注数据 ) Formula (6)

[0125] Among them, Dist i，j This refers to the enhanced formatted edit distance, where f is a constant and depth refers to the depth of the text nested entities in the data to be labeled.

[0126] It should be understood that strengthening the formatted edit distance by the depth of text entity nesting can make the similarity judgment adapt to the complexity of the text structure. When the depth of text entity nesting is large, it means that the entities in the text contain more hierarchical relationships and have a more complex structure, making it easier for differences in expression to occur during annotation. At this time, in formula (6), (1+f×depth) 待标注数据 The coefficient of ) will increase, and the enhanced Dist i, This also increases the requirement for consistency in annotation of complex text. Conversely, when the nesting depth is small, the text structure is simple, annotation descriptions are easier to unify, the coefficient is small, and the adjustment range for similarity is more moderate.

[0127] S402. If the fourth candidate label is a single label and the label score of the fourth candidate label is less than the fifth threshold, then the fourth candidate label is determined as the target label.

[0128] The fifth threshold is determined based on the nesting depth of the text entities in the data to be labeled.

[0129] The candidate tag with the smallest sum of edit distances is selected as the fourth candidate tag because the smaller the sum of edit distances, the higher the text similarity between the candidate tag and other candidate tags, which may indicate that the annotations are similar. When the sum is the smallest, it indicates that the candidate tag has certain differences in text description compared to other candidate tags, which can avoid the situation where it is difficult to distinguish multiple candidate tags due to their high text similarity. At the same time, the fifth threshold is determined according to the depth of text entity nesting, which can adapt to text scenarios with different levels of complexity.

[0130] For example, the calculation process of the fifth threshold can be expressed as follows:

[0131] θ5=ε-(1-ε)×depth 待标注数据 Formula (7)

[0132] Where θ5 represents the fifth threshold and ε represents the fixed threshold.

[0133] As shown in steps S401-S402, the fourth candidate tag is first selected based on the sum of edit distances, and the fifth threshold is determined based on the text entity nesting depth. Normalization is used to eliminate the influence of text length on edit distance, and the formatted edit distance is strengthened by the text entity nesting depth. This allows the selection criteria to adapt to the complexity of the text structure; that is, a large text nesting depth increases the requirement for annotation consistency, while a small nesting depth results in a more moderate adjustment. This ensures that the selected fourth candidate tags meet the scenario requirements in terms of overall text similarity. Combining this with subsequent judgment conditions to determine the target tag allows the final result to reflect both the surface similarity of the text and its inherent structural characteristics, improving the accuracy and rationality of tag selection in different text scenarios.

[0134] The above describes the label fitting process for candidate labels of three different label types and the final target label determined.

[0135] In some embodiments, if the above conditions are not met, the computing device may further perform conflict arbitration on the multiple first candidate labels, the second candidate label with the average of the maximum and second maximum intersection-union ratios, and the multiple fourth candidate labels; that is, the method further includes step S501:

[0136] S501. Under the preset arbitration conditions, conduct conflict arbitration on the target candidate label.

[0137] The target candidate label is any one of the first candidate label, the second candidate label, and the fourth candidate label.

[0138] In some embodiments, when arbitrating conflicts of target candidate tags, the arbitrator can output the arbitration tag, and then automatically arbitrate based on the arbitration rules of the arbitration tag and preset values. In this case, the process is as follows: Figure 5 As shown, the specific steps include the following:

[0139] S601. Obtain the arbitration label for the data to be labeled.

[0140] The arbitration label is obtained by the arbitrators from the data to be labeled.

[0141] Arbitrators are designated personnel who specialize in resolving conflicts through arbitration.

[0142] S602. For each first candidate label, calculate the text similarity between each first candidate label and the arbitration label. If the text similarity is greater than the similarity threshold and the confidence of the arbitration label is greater than the first arbitration threshold, determine the arbitration label as the target label of the data to be labeled.

[0143] Text similarity refers to the degree of similarity between two texts in terms of semantic content and expression. Arbitration labels are usually standard labels that have been manually verified with high authority. The degree of matching between each candidate label and the arbitration label is quantified by text similarity measurement. If the similarity between each candidate label and the arbitration label is greater than a similarity threshold, it means that the arbitration label is similar to the first candidate label. Therefore, the arbitration label can be used as the target label for the data to be labeled.

[0144] In addition, during the arbitration process, the computing device also needs to calculate the confidence level of the arbitration tag and determine whether the arbitration tag is reliable by using the first arbitration threshold.

[0145] In some embodiments, the confidence level of the arbitration label may be obtained by having another arbitrator evaluate the arbitration label.

[0146] For example, the similarity threshold can be 0.7, 0.8, or 0.9. The specific value of the similarity threshold depends on the calculation method of the reference similarity, which will not be elaborated here. The first arbitration threshold can be 0.85, 0.9, or 0.95. This application embodiment does not limit the specific values ​​of the similarity threshold and the first arbitration threshold.

[0147] S603. For the second candidate label, calculate the maximum offset distance between each second candidate label and the arbitration label. If the maximum offset distance is less than the offset threshold and the confidence level of the arbitration label is greater than the second arbitration threshold, determine the arbitration label as the target label of the data to be labeled.

[0148] Offset distance refers to the positional deviation of the detection box of the second candidate label from the detection box of the arbitration label in the horizontal (lateral) and vertical (longitudinal) directions. Maximum offset distance refers to the maximum value between the horizontal and vertical offset distances mentioned above, reflecting the proximity of the detection boxes of the second candidate label and the arbitration label. Therefore, if the maximum offset distance of each second candidate label is less than the offset threshold, it indicates that the detection boxes of the arbitration label and the second candidate labels are consistent, and the arbitration label can be used as the target label for the data to be labeled.

[0149] In some embodiments, the offset threshold can be determined based on the length of the data (image) to be labeled in the corresponding direction (e.g., horizontal length and vertical width).

[0150] For example, the offset threshold can be 5%, 6%, or 7% of the width of the data (image) to be labeled, and the second arbitration threshold can be 0.8, 0.85, 0.9, etc. This application embodiment does not limit the specific values ​​of the offset threshold and the second arbitration threshold.

[0151] S604. For the fourth candidate label, calculate the minimum edit distance between each fourth candidate label and the arbitration label. If the minimum edit distance is less than the edit distance threshold and the confidence of the arbitration label is greater than the third arbitration threshold, determine the arbitration label as the target label of the data to be labeled.

[0152] As mentioned above, the minimum offset distance measures the minimum deviation between two detection boxes in position, reflecting their proximity in a certain direction. Therefore, if the minimum edit distance between each third candidate label and the arbitration label is less than the edit distance threshold, it indicates that the arbitration label is similar to each third candidate label, and thus the arbitration label can be selected as the target label.

[0153] For example, the edit distance threshold can be 1, 2, 3, etc. The third arbitration threshold can specifically be 0.9, 0.91, 0.92, etc. This application embodiment does not limit the specific values ​​of the edit distance threshold and the third arbitration threshold.

[0154] S605. If the arbitration label is not determined as the target label for the data to be labeled, the data to be labeled shall be relabeled.

[0155] Specifically, the process of re-annotating the data to be labeled can be referred to in steps S101-S103, which will not be elaborated here.

[0156] As can be seen from the aforementioned steps S601-S605, this method can automatically arbitrate using arbitration rules, reducing the workload of manual arbitration. Furthermore, manual arbitration is only required when the arbitration rules are insufficient to determine the target label.

[0157] In some embodiments, the process of obtaining candidate labels through annotation models and annotators, such as Figure 6 As shown, the specific steps include the following:

[0158] S701. Determine the corresponding annotation model based on the data type of the data to be annotated.

[0159] One possible implementation is to use the YOLOv7 model for text data and the RoBERTa base model for image data.

[0160] S702. Input the data to be labeled into the labeling model to obtain the labeling results and the corresponding confidence level.

[0161] One possible approach is to first determine the labeling model corresponding to the data type of the data to be labeled, and then input the data to be labeled and the label type to be labeled into the labeling model to obtain the labeling results and the corresponding confidence scores.

[0162] Understandably, by using annotation models to pre-generate annotation results for the data to be annotated, the repetitive annotation work of annotators can be reduced, thus lowering the workload and time cost of manual annotation. At the same time, the annotation results and confidence levels generated by the model can provide annotators with a reference benchmark, helping them to quickly determine the focus of annotation, which can significantly improve the overall annotation efficiency, especially in large-scale data annotation scenarios.

[0163] S703. Determine the annotation mode based on the confidence level corresponding to the annotation results.

[0164] The annotation modes include corrective annotation mode and full annotation mode. Corrective annotation mode instructs annotators to correct the annotation results. Full annotation mode instructs annotators to re-annotate the data to be annotated.

[0165] The pre-annotation process of the annotation model provides an annotator with a reference benchmark for annotation results. Combining this with confidence level-based annotation mode determination enables precise allocation of annotation resources. Specifically, when the confidence level of the annotation results is high (e.g., above a preset threshold), it indicates strong reliability of the model pre-annotation. In this case, a correction annotation mode is triggered—annotators do not need to re-annotate; they only need to check and make local corrections to the pre-annotation results, significantly reducing repetitive work. When the confidence level is low, it indicates limited reference value of the pre-annotation results. In this case, a full annotation mode is activated, requiring annotators to complete the annotation from scratch, avoiding errors caused by relying on low-quality pre-annotations.

[0166] S704. Use the annotation mode to obtain multiple candidate labels for the data to be annotated.

[0167] One possible implementation is that the annotators can annotate the data to be annotated according to the annotation pattern to obtain multiple candidate labels.

[0168] The computing device can determine the annotators who will annotate the data by scoring the annotation effect of each annotator.

[0169] For example, the calculation process for the annotation effectiveness score of the annotator can be expressed as follows:

[0170]

[0171] Where μ refers to the update decay factor of the annotation effect score, and M total This refers to the number of arbitration tags generated by the labeler j as an arbitrator, M. correct This refers to the number of arbitration labels generated by the labeler j as the arbitrator that are identified as target labels.

[0172] As can be seen from steps S701-S704, this method adopts a collaborative annotation mode in which the model undertakes basic annotation and the human focuses on key corrections. This reduces the repetitive work of annotators on simple, highly deterministic data, ensures the annotation quality of low-confidence data through human intervention, and reduces the annotation inconsistency caused by human subjective differences. While improving the overall annotation efficiency, it balances cost and result reliability.

[0173] In some embodiments, after the target label is determined for the data to be labeled, training data can be constructed based on the target label and the data to be labeled, the labeling results of the labeling model and the data to be labeled, and the corresponding type of labeling model can be fine-tuned to optimize the labeling capability of the labeling model.

[0174] In some embodiments, this application also provides a schematic diagram of the application process of a data annotation method, such as... Figure 7 As shown, the specific application process is as follows:

[0175] 1. Pre-labeling.

[0176] This refers to using a labeling model to pre-label the data to be labeled and obtain the labeling results.

[0177] 2. The annotator corrected the annotation.

[0178] This refers to the process of revising and annotating the labels obtained from intelligent pre-annotation by multiple annotators, resulting in multiple candidate labels. Revising the annotation refers to revising the annotation mode.

[0179] 3. Candidate label fitting.

[0180] This refers to determining a fitting scheme based on the label types of multiple candidate labels. It also involves applying the fitting scheme to multiple candidate labels (such as...) Figure 7 The candidate labels 1, 2, and 3 in the dataset are fitted.

[0181] 4. Update the fitting threshold.

[0182] This refers to adjusting the fitting threshold based on the data characteristics of the data to be labeled.

[0183] 5. Determine the fitting results.

[0184] This refers to the fitting result after fitting multiple candidate labels using a fitting threshold. If the fitting result fails, steps 6-8 must be executed first, followed by step 9. If the fitting result is successful, step 9 can be executed directly.

[0185] 6. Conflict classification and handling.

[0186] If the fitting result fails, conflict resolution is performed on multiple candidate labels. Conflict resolution includes automatic correction and manual arbitration.

[0187] 7. Automatic correction.

[0188] This refers to comparing the similarity between the arbitration label provided by the arbitrator and the candidate labels, and determining the arbitration label as the target label based on the comparison results.

[0189] 8. Artificial sanctions.

[0190] This refers to the manual relabeling of items when the automatic correction fails to determine the target label.

[0191] 9. Quality inspection and acceptance.

[0192] If the fitting result is successful, or if the target label for the data to be labeled is determined through automatic correction or manual arbitration, a quality inspection is performed on the labeling result and the target label. The quality inspection is used to determine whether the target label conforms to the format requirements of the training data that support model training.

[0193] 10. Feedback on annotation results.

[0194] This includes updating annotator information and fine-tuning the annotation model.

[0195] 11. Update personnel information.

[0196] This refers to updating the annotation accuracy and annotation effect score of the annotators based on the target labels.

[0197] 12. Fine-tuning of the annotation model.

[0198] This refers to fine-tuning the annotation model based on the data to be labeled and the target labels.

[0199] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the data annotation device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0200] In an exemplary embodiment, this application also provides a data annotation apparatus. This data annotation apparatus may include one or more functional modules for implementing the data annotation method of the above method embodiments.

[0201] For example, such as Figure 8 As shown, the data annotation device includes an acquisition module 810 and a processing module 820.

[0202] The acquisition module 810 is used to acquire multiple candidate labels for the data to be labeled.

[0203] The processing module 820 is used to determine a label fitting scheme based on the label types of multiple candidate labels and the data characteristics of the data to be labeled; the label types include classification labels, detection box labels, and text labels. Each label type corresponds to a data fitting scheme. Based on the label fitting scheme, multiple candidate labels are fitted to determine the target label for the data to be labeled.

[0204] One possible implementation involves multiple candidate labels of category type. The data characteristics of the data to be labeled include the number of categories for the multiple candidate labels. Specifically, processing module 820 selects the candidate label with the highest label score from the multiple candidate labels as the first candidate label. The label score refers to the sum of the labeling accuracy rates of all labelers who labeled the same candidate label. The ratio of the label score of the first candidate label to the sum of the labeling accuracy rates of all labelers is used as the confidence level of the first candidate label. If the first candidate label is a single label and its confidence level is greater than a confidence threshold, the first candidate label is determined as the target label. The confidence threshold is the larger of a first threshold and a second threshold. The first threshold is a preset fixed value, and the second threshold is determined based on the number of categories for the multiple candidate labels and the average label score of the multiple candidate labels.

[0205] One possible implementation involves multiple candidate labels, each with a detection box label type. The data features of the data to be labeled include the image area and the number of targets to be detected. According to the label fitting scheme, the processing module 820 is specifically used to select, from the multiple candidate labels, the candidate label with the largest average detection box intersection-union ratio (IU / U) that is greater than a third threshold as the second candidate label. The average IU / U of a candidate label refers to the average IU / U of the candidate label with each of the other candidate labels. The third threshold is determined based on the image area and the number of targets to be detected in the data to be labeled. If the second candidate label is a single label and the average IU / U of the second candidate label meets a preset condition, the second candidate label is determined as the target label. The preset condition includes: the ratio of the average IU / U of the second candidate label to the average IU / U of the third candidate label is greater than a fourth threshold. The third candidate label is the label whose average IU / U of the second candidate label is second only to the second candidate label.

[0206] One possible implementation involves multiple candidate labels of text type. The data characteristics of the data to be labeled include the nesting depth of the text entities. Specifically, the processing module 820 selects the candidate label with the lowest edit score from the multiple candidate labels as the fourth candidate label. The edit score of a candidate label refers to the sum of the edit distances between that candidate label and other candidate labels. Edit distance refers to the number of times a candidate label is changed to another candidate label. If the fourth candidate label is a single label and its label score is less than a fifth threshold, the fourth candidate label is determined as the target label. The fifth threshold is determined based on the nesting depth of the text entities in the data to be labeled.

[0207] One possible implementation involves conflict arbitration of the target candidate label under preset arbitration conditions. The target candidate label can be any one of the first candidate label, the second candidate label, and the third candidate label.

[0208] One possible implementation involves the processing module 820 specifically acquiring arbitration labels for the data to be labeled. These arbitration labels are obtained by arbitrators annotating the data. For first candidate labels, the text similarity between each first candidate label and the arbitration label is calculated. If the text similarity is greater than a similarity threshold and the confidence level of the arbitration label is greater than a first arbitration threshold, the arbitration label is determined as the target label for the data to be labeled. For second candidate labels, the maximum offset distance between each second candidate label and the arbitration label is calculated. If the maximum offset distance is less than an offset threshold and the confidence level of the arbitration label is greater than a second arbitration threshold, the arbitration label is determined as the target label for the data to be labeled. For third candidate labels, the minimum edit distance between each third candidate label and the arbitration label is calculated. If the minimum edit distance is less than an edit distance threshold and the confidence level of the arbitration label is greater than a third arbitration threshold, the arbitration label is determined as the target label for the data to be labeled. If the arbitration label is not determined as the target label for the data to be labeled, manual arbitration is performed on the data to be labeled.

[0209] In one possible implementation, the processing module 820 is further used to determine the corresponding annotation model based on the data type of the data to be labeled. The data to be labeled is input into the annotation model to obtain the annotation results and corresponding confidence levels. Based on the confidence levels corresponding to the annotation results, the annotation mode is determined. The annotation modes include a corrected annotation mode and a full annotation mode. The corrected annotation mode instructs the annotators to correct the annotation results. The full annotation mode instructs the annotators to re-annotate the data to be labeled. Multiple candidate labels for the data to be labeled are obtained using the annotation modes. It should be noted that... Figure 8 The module division shown is illustrative and represents only one logical functional division; in actual implementation, other division methods are possible. For example, two or more functions can be integrated into a single processing module. These integrated modules can be implemented either in hardware or as software functional modules.

[0210] In exemplary embodiments, as described above, the computing device may specifically be an electronic device with computing processing capabilities, such as a computer or service. In this case, embodiments of this application also provide an electronic device. Figure 9 This is a schematic diagram illustrating the composition of an electronic device provided in an embodiment of this application. For example... Figure 9 As shown, the electronic device includes: a processor 10, a memory 20, a communication line 30, a communication interface 40, and an input / output interface 50.

[0211] The processor 10, memory 20, communication interface 40, and input / output interface 50 can be connected via communication line 30.

[0212] The processor 10 is used to execute instructions stored in the memory 20 to implement the data annotation method provided in the above embodiments of this application. The processor 10 can be a CPU, a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller (MCU) / single-chip microcomputer / microcontroller, a programmable logic device (PLD), or any combination thereof. The processor 10 can also be any other device with processing capabilities, such as a circuit, device, or software module; this application embodiment does not limit this. In one example, the processor 10 may include one or more CPUs, for example... Figure 9 CPU0 and CPU1 are mentioned. As an optional implementation, the electronic device may include multiple processors; for example, in addition to processor 10, it may also include processor 60. Figure 9 (The example shown is a dashed line).

[0213] The memory 20 is used to store instructions. For example, the instructions may be computer programs. Optionally, the memory 20 may be a read-only memory (ROM) or other types of static storage devices that can store static information and / or instructions; it may also be a random access memory (RAM) or other types of dynamic storage devices that can store information and / or instructions; it may also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, etc. The embodiments of this application do not limit this.

[0214] It should be noted that the memory 20 can exist independently of the processor 10 or it can be integrated with the processor 10. The memory 20 can be located inside or outside the electronic device, and this application embodiment does not impose any restrictions on this.

[0215] Communication line 30 is used to transmit information between the components included in the electronic device.

[0216] Communication interface 40 is used to communicate with other devices or other communication networks. These other communication networks can be Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc. Communication interface 40 can be a module, circuit, transceiver, or any device capable of enabling communication.

[0217] Input / output interface 50 is used to enable human-computer interaction between users and electronic devices. For example, it enables action interaction or information exchange between users and electronic devices.

[0218] For example, the input / output interface 50 can be a mouse, keyboard, display screen, or touch screen. Action or information interaction between the user and the electronic device can be achieved through a mouse, keyboard, display screen, or touch screen.

[0219] It should be noted that, Figure 9 The structures shown do not constitute a limitation on electronic devices, except... Figure 9 In addition to the components shown, electronic devices may include more or fewer components than illustrated, or combinations of certain components, or different component arrangements.

[0220] In an exemplary embodiment, this application also provides a readable storage medium including software instructions that, when run on an electronic device, cause the electronic device to perform any of the methods provided in the above embodiments.

[0221] In an exemplary embodiment, this application also provides a computer program product containing computer execution instructions, which, when run on an electronic device, causes the electronic device to perform any of the methods provided in the above embodiments.

[0222] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer-executable instructions. When these computer-executable instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer-executable instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer-executable instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks, SSDs).

[0223] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0224] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

[0225] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data annotation method, characterized in that, The method includes: Obtain multiple candidate labels for the data to be labeled; A label fitting scheme is determined based on the label types of the multiple candidate labels and the data characteristics of the data to be labeled; the label types include classification labels, detection box labels, and text labels; one label type corresponds to one data fitting scheme; According to the label fitting scheme, the multiple candidate labels are fitted to determine the target label of the data to be labeled.

2. The method according to claim 1, characterized in that, The multiple candidate labels are classified labels; the data features of the data to be labeled include the number of classification categories of the multiple candidate labels of the data to be labeled. The step of fitting the multiple candidate labels according to the label fitting scheme to determine the target label of the data to be labeled includes: From the multiple candidate labels, the candidate label with the highest label score is selected as the first candidate label; the label score refers to the sum of the labeling accuracy rates of all labelers who labeled the same candidate label; The confidence level of the first candidate label is the ratio of the label score of the first candidate label to the sum of the labeling accuracy of all the labelers. If the first candidate label is a single label and the confidence level of the first candidate label is greater than the confidence level threshold, the first candidate label is determined to be the target label; the confidence level threshold is the maximum threshold between the first threshold and the second threshold; the first threshold is a preset fixed value, and the second threshold is determined based on the number of classification categories of the multiple candidate labels and the average label score of the multiple candidate labels.

3. The method according to claim 1, characterized in that, The label type of the multiple candidate labels is a detection box label; the data features of the data to be labeled include the image area of ​​the data to be labeled and the number of targets to be detected in the data to be labeled; The step of fitting the multiple candidate labels according to the label fitting scheme to determine the target label of the data to be labeled includes: From the plurality of candidate labels, the candidate label with the largest average cross-union ratio (CUI) of detection boxes that is greater than the third threshold is selected as the second candidate label; the average CUI of a candidate label refers to the average of the CUI of that candidate label with each of the other candidate labels; the third threshold is determined based on the image area of ​​the data to be labeled and the number of targets to be detected in the data to be labeled. If the second candidate label is a single label and the average cross-union ratio (CUI) of the detection frames of the second candidate label meets a preset condition, the second candidate label is determined to be the target label. The preset condition includes: the ratio of the average CUI of the detection frames of the second candidate label to the average CUI of the detection frames of the third candidate label is greater than a fourth threshold; the third candidate label is a label whose average CUI is less than that of the second candidate label.

4. The method according to claim 1, characterized in that, The multiple candidate labels are text labels; the data features of the data to be labeled include the text entity nesting depth of the data to be labeled; the step of fitting the multiple candidate labels according to the label fitting scheme to determine the target label of the data to be labeled includes: From the multiple candidate tags, the candidate tag with the smallest edit score is selected as the fourth candidate tag; the edit score of a candidate tag refers to the sum of the edit distances between the candidate tag and other candidate tags; the edit distance refers to the number of times a candidate tag is changed to another candidate tag. If the fourth candidate label is a single label and the label score of the fourth candidate label is less than the fifth threshold, the fourth candidate label is determined as the target label; the fifth threshold is determined based on the text entity nesting depth of the data to be labeled.

5. The method according to any one of claims 2-4, characterized in that, Under preset arbitration conditions, conflict arbitration is performed on the target candidate label; the target candidate label is any one of the first candidate label, the second candidate label, and the fourth candidate label.

6. The method according to claim 5, characterized in that, Conflict arbitration of target candidate labels includes: Obtain the arbitration label of the data to be labeled; the arbitration label is obtained by the arbitrators from labeling the data to be labeled. For the first candidate label, calculate the text similarity between each first candidate label and the arbitration label. If the text similarity is greater than the similarity threshold and the confidence of the arbitration label is greater than the first arbitration threshold, determine the arbitration label as the target label of the data to be labeled. For the second candidate label, calculate the maximum offset distance between each second candidate label and the arbitration label. If the maximum offset distance is less than the offset threshold and the confidence of the arbitration label is greater than the second arbitration threshold, determine the arbitration label as the target label of the data to be labeled. For the fourth candidate label, calculate the minimum edit distance between each fourth candidate label and the arbitration label. If the minimum edit distance is less than the edit distance threshold and the confidence of the arbitration label is greater than the third arbitration threshold, determine the arbitration label as the target label of the data to be labeled. If the arbitration label is not determined as the target label for the data to be labeled, the data to be labeled shall be relabeled.

7. The method according to claim 1, characterized in that, Before obtaining multiple candidate labels for the data to be labeled, the method further includes: Determine the corresponding annotation model based on the data type of the data to be annotated; The data to be labeled is input into the labeling model to obtain the labeling results and the corresponding confidence scores; The annotation mode is determined based on the confidence level corresponding to the annotation results; the annotation mode includes a corrected annotation mode and a full annotation mode; the corrected annotation mode instructs the annotators to correct the annotation results; the full annotation mode instructs the annotators to re-annotate the data to be annotated; Multiple candidate labels for the data to be labeled are obtained using the aforementioned labeling mode.

8. A data annotation device, characterized in that, The device includes: an acquisition module and a processing module; The acquisition module is used to acquire multiple candidate labels for the data to be labeled; The processing module is used to determine a label fitting scheme based on the label types of the multiple candidate labels and the data characteristics of the data to be labeled; the label types include classification labels, detection box labels, and text labels; one label type corresponds to one data fitting scheme; and the multiple candidate labels are fitted according to the label fitting scheme to determine the target label of the data to be labeled.

9. An electronic device, characterized in that, include: Processor and memory; The memory stores instructions that the processor can execute; When the processor is configured to execute the instructions, the electronic device performs the method as described in any one of claims 1-7.

10. A readable storage medium, characterized in that, include: Software instructions; When the software instructions are executed in the electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-7.

11. A computer program product, characterized in that, include: Computer instructions; When the computer instructions are executed in the electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-7.

Images