Data labeling method and device, electronic device, and storage medium

Abstract

A data labeling method and device, electronic device, and storage medium, the method comprising: labeling the data to be labeled by a first type of labeling model to obtain a first labeling result, wherein the first type of labeling model is constructed based on a first label set; labeling the data to be labeled by a second type of labeling model to obtain a second labeling result, wherein the second type of labeling model is constructed based on a second label set, the first label set and the second label set are both subsets of a target label set corresponding to the data to be labeled, and the first label set is different from the second label set; and fusing the above-mentioned first labeling result and the second labeling result to obtain a target labeling result that matches the data to be labeled. The implementation of the embodiments of the present application can realize fast and low-cost automatic data labeling, which is conducive to improving the efficiency of data labeling.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data labeling method and device, an electronic device, and a storage medium. Background Art

[0002] Currently, the realization of most artificial intelligence technologies is inseparable from supervised learning algorithms, which usually require a large amount of labeled data. Traditional data labeling is mainly based on manual labeling. However, in practice, it is found that such data labeling methods are not only slow, but also require a lot of manpower costs, resulting in a decrease in data labeling efficiency. Summary of the invention

[0003] The embodiments of the present application disclose a data labeling method and device, an electronic device, and a storage medium, which can realize fast and low-cost automatic data labeling, and are conducive to improving the efficiency of data labeling.

[0004] The first aspect of the embodiment of the present application discloses a data labeling method, including:

[0005] Annotating the data to be annotated by using a first type of annotation model to obtain a first annotation result, wherein the first type of annotation model is constructed based on a first label set;

[0006] Annotating the data to be annotated by using a second type of annotation model to obtain a second annotation result, wherein the second type of annotation model is constructed based on a second label set, the first label set and the second label set are both subsets of a target label set corresponding to the data to be annotated, and the first label set is different from the second label set;

[0007] The first labeling result and the second labeling result are merged to obtain a target labeling result that matches the data to be labeled.

[0008] A second aspect of an embodiment of the present application discloses a data labeling device, including:

[0009] A first data labeling unit, configured to label the data to be labeled by using a first type of labeling model to obtain a first labeling result, wherein the first type of labeling model is constructed based on a first label set;

[0010] a second data labeling unit, configured to label the data to be labeled by using a second type of labeling model to obtain a second labeling result, wherein the second type of labeling model is constructed based on a second label set, the first label set and the second label set are both subsets of a target label set corresponding to the data to be labeled, and the first label set is different from the second label set;

[0011] A fusion unit is used to fuse the first annotation result and the second annotation result to obtain a target annotation result that matches the data to be annotated.

[0012] A third aspect of an embodiment of the present application discloses a terminal device, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor implements all or part of the steps in any data labeling method disclosed in the first aspect of the embodiment of the present application.

[0013] A fourth aspect of an embodiment of the present application discloses a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, all or part of the steps in any one of the data labeling methods disclosed in the first aspect of the embodiment of the present application are implemented.

[0014] Compared with the related art, the embodiments of the present application have the following beneficial effects:

[0015] In the embodiment of the present application, two types of annotation models can be set for the data to be annotated to be annotated, wherein the first type of annotation model can be constructed based on the first label set, and the second type of annotation model can be constructed based on the second label set, and the first label set and the second label set are both subsets of the target label set corresponding to the data to be annotated, and the first label set can be distinguished from the second label set. By annotating the data to be annotated with the above-mentioned first type of annotation model, a first annotation result can be obtained; at the same time, by annotating the data to be annotated with the above-mentioned second type of annotation model, a second annotation result can be obtained; by fusing the first annotation result and the second annotation result, a target annotation result matching the data to be annotated can be obtained. It can be seen that by implementing the embodiment of the present application, the data to be annotated can be annotated with two types of data annotation models constructed under different label sets, and then the annotation results obtained by each can be merged to comprehensively judge the data to be annotated with the two types of data annotation models, and obtain a relatively reliable data annotation result, so as to replace the manual data annotation method, realize fast and low-cost automatic data annotation, which is conducive to improving the efficiency of data annotation, and also ensures the accuracy of automatic data annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1is a schematic diagram of a data annotation framework disclosed in an embodiment of the present application;

[0018] Figure 2 It is a flowchart of a data labeling method disclosed in an embodiment of the present application;

[0019] Figure 3 It is a flowchart of another data labeling method disclosed in an embodiment of the present application;

[0020] Figure 4 It is a flowchart of another data labeling method disclosed in an embodiment of the present application;

[0021] Figure 5 is a schematic diagram of a label set generation framework disclosed in an embodiment of the present application;

[0022] Figure 6 It is a flowchart of another data labeling method disclosed in an embodiment of the present application;

[0023] Figure 7 is a schematic diagram of another data annotation framework disclosed in an embodiment of the present application;

[0024] Figure 8 It is a modular schematic diagram of a data labeling device disclosed in an embodiment of the present application;

[0025] Fig. 9 It is a modular schematic diagram of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] It should be noted that the terms "including" and "having" in the embodiments of the present application and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] The embodiments of the present application disclose a data labeling method and device, an electronic device, and a storage medium, which can realize fast and low-cost automatic data labeling, and are conducive to improving the efficiency of data labeling.

[0029] The following is a detailed description with reference to the accompanying drawings.

[0030] See also Figure 1 , Figure 1 1 is a schematic diagram of a data annotation framework disclosed in an embodiment of the present application, including an annotation module 10 and a fusion module 20. Among them, the annotation module 10 may include multiple models for data annotation, exemplarily, it may include a first type of annotation model 11 and a second type of annotation model 12. The above-mentioned first type of annotation model 11 and the second type of annotation model 12 can be used to perform independent data annotation on the data to be annotated, and the fusion module 20 can be used to fuse the annotation results output by the first type of annotation model 11 and the second type of annotation model 12, respectively, to obtain a target annotation result corresponding to the data to be annotated.

[0031] The above-mentioned data labeling refers to adding certain data labels to the data to be labeled, so that the types and attributes of different data can be distinguished according to the data labels, and the data labels can also be used as the identifiers of different data themselves. Specifically, the data to be labeled can be labeled by a certain set of labels, that is, the data to be labeled is classified under a certain data label included in the label set as the labeled data corresponding to the data label. For example, when the data to be labeled is labeled according to the type of data format, the data to be labeled with different data formats can be labeled as audio media format, image media format, document format, etc.; for another example, when the image data to be labeled is labeled according to the different image recognition results, different image data can be labeled as "puppy", "chair", "umbrella", etc.; for another example, when the data to be labeled is labeled according to the difference of the data to be labeled itself, different identifiers (such as digital identifiers, character identifiers, hash value identifiers, etc.) can be used as data labels, and the data to be labeled can be labeled as "1", "2", "3", etc. in sequence. The labeled labeled data can be used in many links such as learning training, recognition and matching in artificial intelligence technology.

[0032] In the embodiment of the present application, when it is necessary to annotate the data to be annotated, the data to be annotated can be annotated by the first type of annotation model 11 and the second type of annotation model 12 respectively based on the above data annotation framework. Among them, the above first type of annotation model 11 can be constructed based on the first label set, and the second type of annotation model 12 can be constructed based on the second label set. Both the first label set and the second label set can be subsets of the target label set corresponding to the data to be annotated, and the first label set can be distinguished from the second label set, so that whether it is through the first type of annotation model 11 or the second type of annotation model 12, the data label corresponding to the data to be annotated can be accurately determined from the data labels included in the target label set, and the data to be annotated can be marked accordingly.

[0033] Exemplarily, taking data annotation on the server as an example, the server can annotate the data to be annotated through the above-mentioned first type of annotation model 11 to obtain a first annotation result; at the same time, the server can annotate the same data to be annotated through the above-mentioned second type of annotation model 12 to obtain a second annotation result; on this basis, the server can merge the above-mentioned first annotation result and the second annotation result to obtain a target annotation result that matches the data to be annotated. Thus, the server can combine the judgments of the two types of data annotation models on the data to be annotated to obtain a relatively reliable data annotation result, ensure the accuracy of data annotation, and effectively replace the slow and costly manual data annotation, realize relatively fast and low-cost automatic data annotation, and help improve the efficiency of data annotation.

[0034] See also Figure 2 , Figure 2 Schematic diagram of a data annotation method disclosed in the embodiment of the present application. Figure 2 As shown, the data annotation method may include the following steps:

[0035] 202. Annotate the data to be annotated by using a first type of annotation model to obtain a first annotation result, wherein the first type of annotation model is constructed based on a first label set.

[0036] Exemplarily, the above-mentioned data to be labeled may include various types of data that need to be identified and labeled by data tags, such as character data, image data, face data, etc. to be labeled, which is not specifically limited in the embodiments of the present application.

[0037] Among them, the above-mentioned data to be labeled can correspond to a target tag set, and the target tag set may include multiple data tags. The data tags included in the target tag set may be set in advance according to the actual data classification requirements. For example, taking the target tag set corresponding to audio data as an example, the data tags included in the target tag set may include various audio keywords; for another example, taking the target tag set corresponding to image data as an example, the data tags included in the target tag set may include various image recognition objects. The data to be labeled can be labeled by the data tags belonging to the target tag set, then the first tag set may belong to a subset of the target tag set, and the above-mentioned first type of labeling model may be constructed based on the first tag set, so that it can be ensured that when the data to be labeled is labeled by the first type of labeling model, the first labeling result obtained belongs to the above-mentioned target tag set. Optionally, when the first tag set is not a true subset of the above-mentioned target tag set, that is, the first tag set is equal to the target tag set, the data to be labeled is labeled by the first type of labeling model, which can avoid the situation that there is no data tag corresponding to the data to be labeled in the first tag set, thereby ensuring that the first labeling result belonging to the above-mentioned target tag set can be obtained.

[0038] In some embodiments, the first type of annotation model may include only one data annotation model, which can be obtained by training the sample data corresponding to the above-mentioned first label set based on a certain classification algorithm. Exemplarily, taking the annotation of text data (such as directly acquired text data, text data indirectly obtained through audio data conversion, etc.) as an example, the first type of annotation model can be trained on text samples carrying data labels in the first label set based on any classification algorithm such as FastText and TextCNN. In other embodiments, the first type of annotation model may also include multiple different data annotation models. Similarly, each data annotation model can be trained on the sample data corresponding to the above-mentioned first label set based on different classification algorithms, thereby ensuring the differences between the first type of annotation models, so that the data to be annotated can be annotated and verified with each other through multiple classification algorithms at the same time, effectively improving the accuracy of data annotation.

[0039] 204. Annotate the data to be annotated by using a second type of annotation model to obtain a second annotation result. The second type of annotation model is constructed based on a second label set. Both the first label set and the second label set are subsets of a target label set corresponding to the data to be annotated, and the first label set is different from the second label set.

[0040] It can be understood that the process of annotating the data to be annotated by the second type of annotation model in step 204 is similar to the process of annotating the data to be annotated by the first type of annotation model in step 202. The second label set may also be a subset of the target label set corresponding to the data to be annotated, and the second type of annotation model may be constructed based on the second label set, so as to ensure that when the data to be annotated is annotated by the second type of annotation model, the second annotation result obtained also belongs to the target label set.

[0041] It is worth noting that the above-mentioned first label set can be distinguished from the second label set, that is, the first label set is not equal to the second label set. Optionally, when the first label set is equal to the target label set, the second label set can be a true subset of the first label set, that is, the second label set belongs to a true subset of the target label set, so that when the data to be labeled is labeled by the second type of labeling model, the data to be labeled can be further repeatedly labeled according to the data labels belonging to the second label set, so that the data to be labeled can be labeled and verified with each other at the same time through different label systems, which further improves the accuracy of data labeling.

[0042] Furthermore, in some embodiments, the second type of annotation model may include only one data annotation model, which may be obtained by training the sample data corresponding to the second label set based on a certain classification algorithm. In other embodiments, the second type of annotation model may also include multiple different data annotation models, each of which may be obtained by training the sample data corresponding to the second label set based on a different classification algorithm.

[0043] The data labeling method disclosed in the embodiments of the present application can be executed by electronic devices or systems such as servers and terminal devices, and is not specifically limited in the embodiments of the present application. By way of example, the following description will be mainly based on the server. In the embodiments of the present application, the server can perform data labeling on the data to be labeled through the above-mentioned first type of labeling model to obtain a first labeling result; at the same time, the server can also perform data labeling on the data to be labeled through the above-mentioned second type of labeling model to obtain a second labeling result. Based on the first labeling result and the second labeling result, the server can fuse them in subsequent steps to obtain a target labeling result that matches and is unified with the data to be labeled.

[0044] 206. The first labeling result and the second labeling result are merged to obtain a target labeling result that matches the data to be labeled.

[0045] In an embodiment of the present application, after the server annotates the data to be annotated using the first type of annotation model and the second type of annotation model respectively and obtains the corresponding first annotation result and the second annotation result, it can also fuse the two according to the first annotation result and the second annotation result to obtain a target annotation result that matches the data to be annotated and is unified.

[0046] In one embodiment, the server may perform a consistency analysis on the first annotation result and the second annotation result by means of a consistency check, and merge the first annotation result and the second annotation result according to the result of the consistency analysis. Exemplarily, taking the annotation of text data as an example, the server may determine whether the first annotation result (such as annotated as text A) and the second annotation result (such as annotated as text B) of the text data to be annotated are consistent. If they are consistent (i.e., A=B), the server may determine either the first annotation result or the second annotation result as the target annotation result; if they are not consistent (i.e., A≠B), the server may perform subsequent annotation correction operations until the target annotation result that passes the consistency check is obtained.

[0047] In another embodiment, the server may also compare the first annotation result and the second annotation result by calculating the similarity, and merge the first annotation result and the second annotation result according to the similarity between the two. Exemplarily, still taking the annotation of text data as an example, the server may calculate the similarity between the first annotation result (A) and the second annotation result (B) of the text data to be annotated. If the similarity is higher than a similarity threshold (such as 90%, 95%, etc.), the server may determine either the first annotation result or the second annotation result as the target annotation result; if it is lower than the similarity threshold, the server may further analyze the first annotation result and the second annotation result respectively, and determine one of them (such as one with higher data annotation confidence, one with higher fault tolerance, etc.) as the target annotation result.

[0048] It can be seen that the data labeling method described in the above embodiment can be implemented to label the data to be labeled by using two types of data labeling models constructed under different label sets respectively, and then the labeling results obtained by each can be integrated to comprehensively judge the data to be labeled by the two types of data labeling models, so as to obtain relatively reliable data labeling results, thereby replacing the method of manually labeling data and realizing fast and low-cost automatic data labeling, which is conducive to improving the efficiency of data labeling, and also ensures the accuracy of automatic data labeling.

[0049] See also Figure 3 , Figure 3 FIG. 1 is a flow chart of another data labeling method disclosed in an embodiment of the present application. Figure 3As shown, the data annotation method may include the following steps:

[0050] 302. Determine a target label set corresponding to the data to be labeled according to a data labeling scenario corresponding to the data to be labeled.

[0051] In the embodiment of the present application, when the server needs to annotate the data to be annotated, it can first determine the data annotation scenario corresponding to the data to be annotated. For example, according to the difference in the functional scenarios of obtaining or using the data to be annotated, different data to be annotated can correspond to different data annotation scenarios, such as image semantic recognition scenarios that require annotating image data, intelligent voice assistant scenarios that require annotating voice data, etc., which are not specifically limited in the embodiment of the present application.

[0052] Exemplarily, the server can determine the data annotation scenario corresponding to the data to be annotated by querying, identifying, interactive communication, etc. For example, the server can query the association between the data to be annotated and the data annotation scenario, such as querying through the keyword list of various data annotation scenarios stored in the server, the list of annotated data of various data annotation scenarios, etc.; for another example, the server can also perform preliminary identification of the data to be annotated, and divide the data annotation scenario corresponding to the data to be annotated according to the preliminary identification result; for another example, if the data to be annotated is obtained through a terminal device, the server can also communicate with the terminal device, and then obtain the data annotation scenario corresponding to the data to be annotated from the terminal device.

[0053] After determining the data annotation scenario corresponding to the data to be annotated, the server can further determine the target tag set corresponding to the data to be annotated based on the data annotation scenario. Exemplarily, if the data annotation scenario already has a corresponding data tag set, the server can directly obtain the data tag set as the target tag set corresponding to the data to be annotated; if the data annotation scenario does not yet have a corresponding data tag set, the server can analyze the data annotation scenario and generate a suitable data tag set as the target tag set corresponding to the data to be annotated.

[0054] For example, in the intelligent voice assistant scenario, the server can determine the various data tags in the scenario (such as keywords and voice features for waking up the voice assistant to implement the corresponding skills) according to the various skills that the voice assistant needs to support (such as setting an alarm, turning on the camera, searching for files, etc.), and then add the data tags corresponding to all skills to the data tag set corresponding to the scenario. Exemplarily, if the voice assistant needs to support N skills (N is a positive integer), the data tag set corresponding to the scenario can include N one-to-one corresponding data tags. When the server needs to annotate the voice data obtained after waking up the voice assistant, the data tag set can be used as the target tag set corresponding to the voice data to be annotated, and then the first tag set and the second tag set can be determined from the target tag set in the subsequent steps, so as to annotate the voice data to be annotated by the data annotation models constructed by different tag sets.

[0055] 304. Determine a first tag set from the target tag set, where the first tag set includes multiple data tags belonging to the target tag set.

[0056] Exemplarily, since the first label set should belong to a subset of the above-mentioned target label set, the server can determine some data labels from the multiple data labels included in the target label set and add them to the first label set; or all data labels included in the target label set can be added to the first label set, that is, a first label set equal to the target label set is generated, so that when the data to be labeled is subsequently labeled by using the first type of labeling model constructed based on the first label set, the situation that there is no data label corresponding to the data to be labeled in the first label set can be avoided, thereby ensuring that the first labeling result belonging to the above-mentioned target label set can be obtained.

[0057] 306. Generate a second tag set according to the first tag set.

[0058] In the embodiment of the present application, the second label set can be different from the first label set. Further, the second label set can be a true subset of the first label set, so that when the data to be labeled is subsequently labeled by the second type of labeling model constructed based on the second label set, the data to be labeled can be further repeatedly labeled according to the data labels belonging to the second label set, so that the data to be labeled can be labeled and verified with each other at the same time through different label systems, thereby further improving the accuracy of data labeling.

[0059] As an optional implementation, when the server generates the second tag set based on the above-mentioned first tag set, it may be based on the similarity between each data tag included in the first tag set, and determine similar data tags to generate the second tag set. Exemplarily, the server may calculate the similarity between each data tag included in the first tag set and each other data tag in the first tag set, and obtain the similarity of multiple groups of data tag pairs; if the similarity of any group of data tag pairs meets the similarity condition, the server may add the data tags included in the data tag pairs whose similarity meets the similarity condition to the second tag set. By implementing the above method, the data to be labeled that may be confused with similar data tags can be labeled again using the data tags that meet the similarity condition included in the second tag set, so that the data to be labeled can be identified and distinguished more carefully, reducing the possibility of the data to be labeled being mislabeled, thereby improving the accuracy of data labeling.

[0060] Among them, the similarity between the above data labels can be calculated based on the attributes of the data labels themselves. For example, in a character recognition scenario, by analyzing and calculating the character labels "sun" and "moon" (such as calculating the overlap of feature points for images containing the corresponding characters, calculating the overlap of strokes for the stroke combinations of the corresponding characters, etc.), it can be obtained that the two have a relatively high similarity; and by analyzing and calculating the character labels "sun" and "mountain", it can be obtained that the two have a relatively low similarity. For another example, when the data labels are uniformly represented in the form of vectors, the calculation of the similarity between the above data labels can also be obtained by calculating the vector similarity between the corresponding data label vectors.

[0061] In one embodiment, the similarity condition may include a sorting condition related to similarity. Exemplarily, after the server calculates the similarity between each data tag and each other data tag in the first tag set for each data tag included in the first tag set, it may also sort the calculated similarities of each group of data tag pairs in order from high to low similarity, and add the data tags included in the data tag pairs whose similarity sorting satisfies the order condition to the second tag set. For example, if the first tag set includes N data tags (which can be represented by 1, 2, 3, ..., N), after the server calculates the similarity between any two data tags, it may add M groups of data tags that are ranked higher in similarity from high to low (i.e., have higher similarity) to the second tag set, where For example, when N=10 and M=5, if the top five groups of data tags ranked by similarity are (1, 2), (5, 7), (2, 9), (7, 8), and (9, 4), the server can select data tags 1, 2, 5, 7, 9, 8, and 4 from the first tag set and add them to the second tag set.

[0062] In another embodiment, the above-mentioned similarity condition may also include a threshold condition related to similarity. Exemplarily, after respectively calculating the similarity between each data tag and each other data tag in the first tag set, the server may also obtain data tag pairs whose similarity is higher than the similarity threshold, and add the data tags contained therein to the second tag set. For example, still taking the first tag set including the above-mentioned N data tags as an example, when the data tags with similarity higher than the similarity threshold include three groups of (1, 2), (5, 7), and (2, 9), the server may select data tags 1, 2, 5, 7, and 9 in the first tag set and add them to the second tag set.

[0063] 308. Input the data to be labeled into the first type of labeling model, label the data to be labeled using the data labels contained in the first type of labeling model, and obtain a first labeling result output by the first type of labeling model, wherein the first type of labeling model is trained based on the first sample data, and the data labels carried by the first sample data belong to the first label set.

[0064] In an embodiment of the present application, when the server annotates the data to be annotated through the first type of annotation model, the data to be annotated can be first input into the first type of annotation model. Optionally, if the first type of annotation model includes multiple data annotation models, the server can input the data to be annotated into each of the first type of annotation models respectively. On this basis, each first type of annotation model can annotate the data to be annotated according to the data tags it contains, thereby obtaining at least one first annotation result.

[0065] It should be noted that before actually annotating the data to be annotated through the first type of annotation model, the first type of annotation model needs to be trained first. Taking the above training process on the server as an example, in order to build the first type of annotation model based on the first label set, the server can first obtain the first sample data carrying the data label belonging to the first label set, and the first sample data is the annotated data that has been annotated with the data label belonging to the first label set. Using the first sample data, the server can select different classification algorithms to train it, and finally obtain the trained first type of annotation model.

[0066] 310. Input the data to be labeled into the second type of labeling model, label the data to be labeled using the data labels contained in the second type of labeling model, and obtain a second labeling result output by the second type of labeling model, wherein the second type of labeling model is trained based on the second sample data, and the data labels carried by the second sample data belong to the second label set.

[0067] Among them, step 310 is similar to the above-mentioned step 308 and will not be repeated here. It should be noted that before actually using the second type of annotation model to annotate the data to be annotated, the second type of annotation model also needs to be trained first. Specifically, the server can first obtain the second sample data carrying the data label belonging to the second label set, and the second sample data is the labeled data that has been annotated with the data label belonging to the second label set. Using the second sample data, the server can select different classification algorithms to train it, and finally obtain a trained second type of annotation model. Optionally, when training the second type of annotation model, the server can select the same classification algorithm as when training the first type of annotation model, thereby reducing the differences between the first type of annotation model and the second type of annotation model except for the label system, avoiding potential interference factors that cause data annotation errors as much as possible, and improving the accuracy and reliability of the fusion of annotation results in subsequent steps.

[0068] 312. Perform a consistency check on the first labeling result and the second labeling result. If the consistency check passes, determine a target labeling result that matches the data to be labeled from the first labeling result or the second labeling result.

[0069] In an embodiment of the present application, the server may perform a consistency analysis on the first annotation result and the second annotation result by means of a consistency check, so as to merge the first annotation result and the second annotation result according to the analysis result of the consistency analysis. For example, taking the annotation of text data as an example, the server may determine whether the first annotation result (such as annotated as text A) and the second annotation result (such as annotated as text B) of the text data to be annotated are consistent. If they are consistent (i.e., A=B), the consistency check passes, and the server may determine either the first annotation result or the second annotation result as the target annotation result.

[0070] It can be seen that the implementation of the data labeling method described in the above embodiment can realize fast and low-cost automatic data labeling, which is conducive to improving the efficiency of data labeling, and also ensures the accuracy of automatic data labeling; in addition, according to the data labeling scenario corresponding to the data to be labeled, the corresponding target label set is determined, and the scope of the label system of the data to be labeled can be accurately defined using the actual scenario, thereby helping to reasonably determine one or more sub-label sets therein, so as to realize the labeling of the data to be labeled under multiple label sets, and further improve the accuracy of data labeling; in addition, for the data to be labeled that may be confused with similar data labels, it is re-labeled using data labels that meet similar conditions, so that the data to be labeled can be more finely identified and distinguished, reducing the possibility of the data to be labeled being mislabeled, and further improving the accuracy of data labeling.

[0071] like Figure 4 As shown, in one embodiment, the above step 306 may include the following steps:

[0072] 3061. For each data label included in the first label set, obtain the labeled data corresponding to each data label.

[0073] Exemplarily, the first tag set may include N (N is a positive integer) data tags, and the labeled data corresponding to each data tag may include M1, M2, ..., M N It can be understood that the amount of labeled data corresponding to each data label may be the same or different.

[0074] 3062. According to the labeled data corresponding to each data label, obtain the label vector corresponding to each data label respectively.

[0075] In an embodiment of the present application, the labeled data corresponding to each data tag should have the same data format, but the data formats of the labeled data corresponding to different data tags may be the same or different. For example, if the first tag set includes multiple different data format tags, then each data format tag included in the first tag set may correspond to labeled data in different data formats such as audio media format, image media format, document format, etc. For another example, if the first tag set includes multiple different data content tags, such as different text tags, then each text tag included in the first tag set may correspond to labeled data in text format.

[0076] For labeled data in different data formats, different embedding technologies can be used to map the labeled data to corresponding data vectors. For example, for text data, algorithms such as word2vec and doc2vec can be used to map text data to corresponding word vectors or text vectors; for image data, the GraphEmbedding algorithm can be used to map high-dimensional dense graph data to low-dimensional dense vectors. On this basis, by pooling all data vectors corresponding to each data label, the label vector uniquely corresponding to each data label can be obtained.

[0077] In one embodiment, after obtaining the labeled data corresponding to each data label in the first label set, the server can map each labeled data to a data vector corresponding to each labeled data for all labeled data corresponding to each data label, and then average pooling can be performed on the data vectors corresponding to all labeled data to obtain a label vector corresponding to each data label. For a data label t in the first label set, the corresponding label vector X is obtained by average pooling. t The formula can be shown as the following formula 1.

[0078] Formula 1:

[0079]

[0080] Among them, a i is the data vector corresponding to each labeled data corresponding to the data label t, M t Then is the amount of labeled data corresponding to the data label t. By averaging all data vectors corresponding to the data label t, we can get the result of average pooling, that is, the label vector X corresponding to the data label t. t .

[0081] 3063. Calculate the vector similarities between the label vector corresponding to each data label and the label vectors corresponding to other data labels in the first label set to obtain the similarities of the corresponding multiple groups of data label pairs.

[0082] For example, when calculating the two label vectors X p and X q When measuring the vector similarity between the two label vectors, we can calculate the cosine value of the angle θ between the two label vectors cos(θ) p,q , and use this to measure the similarity of the data label pairs corresponding to the two label vectors. Among them, cos(θ) p,q The smaller it is (the minimum is -1), the smaller the label vector X is. p and Xq The smaller the similarity between vectors, the smaller the similarity between corresponding data label pairs; cos(θ) p,q The larger the value (maximum is 1), the greater the label vector X p and X q The greater the similarity between the vectors, the greater the similarity between the corresponding data label pairs.

[0083] In the embodiment of the present application, for each data label in the first label set, the label vector X p , the server can calculate the label vector X corresponding to each other data label in the first label set q The vector similarity S between p,q , the calculation formula can be shown as the following formula 2.

[0084] Formula 2:

[0085]

[0086] Among them, ||X p || and ||X q ||represents the label vector X p and X q 2-norm of the label vector X p and X q N is the number of data labels included in the first label set; m represents the dimension of the corresponding label vector (determined by the embedding technology used when mapping the labeled data to the corresponding data vector in step 3062, such as 100, 150, etc.). It can be understood that the label vector X p and X q The cosine value of the angle θ between them is cos(θ) p,q It is only a way to measure the similarity of the data label pairs corresponding to the two label vectors. In some embodiments, the similarity of the corresponding data label pairs can also be obtained by calculating the sine value of the angle between the two label vectors, or directly calculating the angle between the two label vectors.

[0087] 3064. Sort the similarities of each group of data label pairs calculated in descending order of similarity.

[0088] 3065. Add the data tags included in the data tag pairs whose similarity sorting satisfies the sequence condition to the second tag set.

[0089] Among them, step 3064 and step 3065 are similar to the embodiment of the above-mentioned step 306 and are not repeated here.

[0090] Please also read Figure 5 , Figure 5 is a schematic diagram of a label set generation framework disclosed in an embodiment of the present application, which can be applied to the process of generating the second label set according to the first label set. Optionally, when the first label set is not equal to the target label set corresponding to the data to be labeled, the label set generation framework can also be applied to the process of generating the first label set and the second label set according to the target label set. Figure 5 As shown, for the labeled data corresponding to the N data labels included in the target label set, the label vectors corresponding to each data label can be obtained by respectively passing through the embedding module 501 and the average pooling module 502 in sequence; then, by calculating the similarity between each label vector in pairs in the similarity calculation module 503, and sorting the similarities calculated by the similarity calculation module 503 in the similarity sorting module 504, the data labels whose similarity sorting meets different order conditions can be added to the first label set 505 and the second label set 506 respectively, thereby realizing the set construction of the first label set 505 and the second label set 506.

[0091] By implementing the above method, it is possible to construct a new label set using data labels that meet similar conditions for the data to be labeled that may be confused with similar data labels, and label the data to be labeled using the label set, thereby enabling more detailed identification and distinction of the data to be labeled, reducing the possibility of the data to be labeled being mislabeled, and helping to further improve the accuracy of data labeling.

[0092] See also Figure 6 , Figure 6 FIG. 1 is a flow chart of another data labeling method disclosed in the embodiment of the present application. Figure 6 As shown, the data annotation method may include the following steps:

[0093] 602. Determine a target label set corresponding to the data to be labeled according to a data labeling scenario corresponding to the data to be labeled.

[0094] 604. Determine a first tag set from the target tag set, where the first tag set includes a plurality of data tags belonging to the target tag set.

[0095] 606. Generate a second tag set according to the first tag set.

[0096] Among them, step 602, step 604 and step 606 are similar to the above-mentioned step 302, step 304 and step 306, and will not be repeated here.

[0097] 608. Input the data to be labeled into the first type of labeling model, label the data to be labeled using the data labels contained in the first type of labeling model, and obtain a first labeling result output by the first type of labeling model, wherein the first type of labeling model is trained based on the first sample data, and the data labels carried by the first sample data belong to the first label set.

[0098] 610. Input the data to be labeled into the second type of labeling model, label the data to be labeled using the data labels contained in the second type of labeling model, and obtain a second labeling result output by the second type of labeling model, wherein the second type of labeling model is trained based on the second sample data, and the data labels carried by the second sample data belong to the second label set.

[0099] Among them, step 608 and step 610 are similar to the above-mentioned step 308 and step 310, and will not be repeated here.

[0100] 612. Compare the first annotation results output by each first annotation model to obtain a first comparison result.

[0101] Among them, the above-mentioned second label set can be a true subset of the first label set, and the first type of annotation model can include multiple first annotation models, and the second type of annotation model can include multiple second annotation models. The above-mentioned multiple first annotation models correspond to different model architectures respectively, and the above-mentioned multiple second annotation models also correspond to different model architectures respectively.

[0102] In an embodiment of the present application, the first type of annotation model may include multiple different first annotation models, so that when the data to be annotated is annotated by the first type of annotation model, at least one first annotation result output by each first annotation model can be obtained. By comparing the above first annotation results, it can be determined whether the various first annotation results are consistent (that is, whether the various first annotation results are the same), and a corresponding first comparison result can be obtained, and the first comparison result can be used to characterize the consistency between the above first annotation results.

[0103] 614. If the first comparison result indicates that the first annotation results output by each first annotation model are the same and the first annotation result does not belong to the second label set, the first annotation result is determined as a target annotation result that matches the data to be annotated.

[0104] 616. If the first annotation result belongs to the second label set, compare the second annotation results output by each second annotation model to obtain a second comparison result.

[0105] Exemplarily, when the above-mentioned first annotation result belongs to the second label set, it means that the data to be annotated may be mislabeled with similar data labels. At this time, if the first comparison result indicates that the first annotation result is consistent (that is, each first annotation result is the same), the server can annotate the data to be annotated again through the second type of annotation model, so that the data to be annotated can be more finely identified and distinguished. It can be understood that the annotation of the annotated data by the above-mentioned first type of annotation model and the second type of annotation model is actually carried out simultaneously, and there is no difference in order. Only when the server merges the first annotation result and the second annotation result, it is necessary to perform a consistency check with a sequence.

[0106] In an embodiment of the present application, the second type of annotation model may also include multiple different second annotation models, so that when the data to be annotated is annotated by the second type of annotation model, at least one second annotation result output by each second annotation model can be obtained. By comparing the above second annotation results, it can be determined whether the second annotation results are consistent (that is, whether the second annotation results are the same), and a corresponding second comparison result can be obtained, which can be used to characterize the consistency between the above second annotation results.

[0107] When the first labeling result does not belong to the second label set, it means that the possibility of the data to be labeled being mislabeled by similar data labels is low. At this time, if the first comparison result indicates that the first labeling result is consistent, the server can directly determine that the consistency check has passed and determine the first labeling result as the target labeling result that matches the data to be labeled.

[0108] 618. If the second comparison result indicates that the second annotation results output by each first annotation model are the same, the second annotation result is determined as a target annotation result that matches the data to be annotated.

[0109] In an embodiment of the present application, if the first comparison result indicates that the first annotation result is consistent, and the second comparison result indicates that the second annotation result is consistent, the server can confirm that the first annotation result and the second annotation result are both consistent, and then determine that the consistency check has passed, and determine the second annotation result as the target annotation result that matches the data to be annotated.

[0110] Please also read Figure 7 , Figure 7 is a schematic diagram of another data annotation framework disclosed in an embodiment of the present application, and Figure 7 The data annotation framework shown is composed of Figure 1 The data annotation framework shown in Figure 1 is optimized. Figure 7As shown, the first type of annotation model 11 may include multiple first annotation models 11a with different model architectures, and the second type of annotation model 12 may include multiple second annotation models 12a with different model architectures. Among them, each of the first annotation models 11a and the second annotation model 12a can be used to perform independent data annotation on the data to be annotated, and the fusion module 20 can be used to fuse the annotation results output by the above-mentioned data annotation models (i.e., perform consistency verification), and finally obtain the target annotation result corresponding to the data to be annotated.

[0111] As an optional implementation, when comparing the annotation results output by the above-mentioned data annotation models, the server may only compare the annotation results that meet certain confidence conditions, thereby preventing the annotation results with lower confidence from reducing the accuracy of the target annotation results obtained after fusion. Exemplarily, the server may obtain the first annotation results with a confidence higher than the first confidence threshold from the first annotation results output by each first annotation model, and compare the first annotation results with a confidence higher than the first confidence threshold to obtain the corresponding first comparison result. At the same time, the server may also obtain the second annotation results with a confidence higher than the second confidence threshold from the second annotation results output by each second annotation model, and compare the second annotation results with a confidence higher than the second confidence threshold to obtain the corresponding second comparison result. By increasing the confidence as the threshold for screening the annotation results, it is helpful to eliminate the annotation results with lower confidence and further improve the accuracy of data annotation for the data to be annotated.

[0112] As another optional implementation, if the above consistency check fails, that is, the first annotation result and the second annotation result corresponding to the data to be annotated do not meet any of the above consistency check passing conditions, the server can iteratively update the above data annotation models to achieve the correction of the first annotation result and the second annotation result. Exemplarily, when the above consistency check fails, the server can obtain a corrected annotation result for the data to be annotated, wherein the corrected annotation result can be obtained by annotation through other data annotation models, or by manual annotation. On this basis, the server can respectively correct the above first type of annotation model and the second type of annotation model according to the data to be annotated and the above corrected annotation result, and re-execute the step of annotating the data to be annotated through the first type of annotation model to obtain the first annotation result, and the step of annotating the data to be annotated through the second type of annotation model to obtain the second annotation result, until the consistency check of the new first annotation result and the second annotation result passes, and the final target annotation result is obtained. Figure 7As shown, the above-mentioned data annotation framework may further include a correction annotation module 30. The correction annotation module 30 may be used to obtain corresponding correction annotation results when the fusion module 20 fails to pass the consistency check of the annotation results output by the above-mentioned data annotation models, that is, fails to obtain the target annotation results corresponding to the data to be annotated, and feed them back to the above-mentioned data annotation models for correction, so as to further improve the accuracy of data annotation.

[0113] It can be seen that implementing the data labeling method described in the above embodiment can achieve fast and low-cost automatic data labeling, which is conducive to improving the efficiency of data labeling, while also ensuring the accuracy of automatic data labeling; in addition, by performing consistency verification on the labeling results output by multiple data labeling models constructed under different label sets, it can be ensured that the target labeling results corresponding to the data to be labeled are obtained only when the labeling results are consistent, thereby further improving the accuracy and reliability of data labeling.

[0114] See also Figure 8 , Figure 8 Schematic diagram of a modular data labeling device disclosed in the embodiment of the present application. Figure 8 As shown, the data annotation device may include a first data annotation unit 801, a second data annotation unit 802, and a fusion unit 803, wherein:

[0115] A first data labeling unit 801 is used to label the data to be labeled by using a first type of labeling model to obtain a first labeling result, wherein the first type of labeling model is constructed based on a first label set;

[0116] A second data labeling unit 802 is used to label the data to be labeled by using a second type of labeling model to obtain a second labeling result, wherein the second type of labeling model is constructed based on a second label set, and both the first label set and the second label set are subsets of a target label set corresponding to the data to be labeled, and the first label set is different from the second label set;

[0117] The fusion unit 803 is used to fuse the first annotation result and the second annotation result to obtain a target annotation result that matches the data to be annotated.

[0118] It can be seen that the data labeling device described in the above embodiment can label the data to be labeled by using two types of data labeling models constructed under different label sets respectively, and then can fuse the labeling results obtained respectively, so as to comprehensively judge the data to be labeled by the two types of data labeling models, and obtain relatively reliable data labeling results, thereby replacing the method of manual labeling of data, realizing fast and low-cost automatic data labeling, which is conducive to improving the efficiency of data labeling, and also ensures the accuracy of automatic data labeling.

[0119] In one embodiment, Figure 8 The data annotation device shown may further include a first determination unit, a second determination unit, and a set generation unit, which are not shown in the figure, wherein:

[0120] A first determining unit, configured to determine a target label set corresponding to the data to be labeled according to a data labeling scenario corresponding to the data to be labeled;

[0121] A second determining unit, configured to determine a first tag set from the target tag set, the first tag set including a plurality of data tags belonging to the target tag set;

[0122] The set generating unit is used to generate a second label set according to the first label set.

[0123] The data labeling device described in the above embodiment can use actual scenarios to accurately define the scope of the label system of the data to be labeled, thereby helping to reasonably determine one or more sub-label sets to achieve labeling of the data to be labeled under multiple label sets, further improving the accuracy of data labeling.

[0124] In one embodiment, the above-mentioned set generation unit may further include a similarity calculation subunit and an adding subunit (not shown), wherein:

[0125] A similarity calculation subunit, for calculating, for each data tag included in the first tag set, the similarity between each data tag and each other data tag in the first tag set;

[0126] The adding subunit is used to add the data tags whose similarity satisfies the similarity condition to the second tag set when the similarity between any two data tags satisfies the similarity condition.

[0127] In one embodiment, the adding unit may be specifically used to sort the similarities between any two data tags in descending order of similarity, and add data tags whose similarity sorting satisfies the order condition to the second tag set.

[0128] By using the data labeling device described in the above embodiment, the data to be labeled that may be confused with similar data labels can be re-labeled using data labels that meet similar conditions. This can more finely identify and distinguish the data to be labeled, reduce the possibility of the data to be labeled being mislabeled, and further improve the accuracy of data labeling.

[0129] In one embodiment, the above-mentioned set generation unit may further include a data acquisition subunit and a vector acquisition subunit (not shown), wherein:

[0130] A data acquisition subunit, configured to acquire, for each data tag included in the first tag set, the labeled data corresponding to each data tag;

[0131] A vector acquisition subunit is used to acquire a label vector corresponding to each data label according to the labeled data corresponding to each data label;

[0132] The similarity calculation subunit can be specifically used to calculate the vector similarity between the label vector corresponding to each data label and the label vectors corresponding to other data labels in the first label set, so as to obtain the similarity between the corresponding data labels.

[0133] In one embodiment, the above-mentioned vector acquisition subunit can be specifically used to map each labeled data into a data vector corresponding to each labeled data for all labeled data corresponding to each data label; and to average pool the data vectors corresponding to all labeled data to obtain a label vector corresponding to each data label.

[0134] The data labeling device described in the above embodiment can construct a new label set using data labels that meet similar conditions for the data to be labeled that may be confused with similar data labels, and label the data to be labeled using the label set, thereby enabling more detailed identification and distinction of the data to be labeled, reducing the possibility of the data to be labeled being mislabeled, and helping to further improve the accuracy of data labeling.

[0135] In one embodiment, the first data labeling unit 801 may be specifically used to input the data to be labeled into a first type of labeling model, label the data to be labeled using the data labels included in the first type of labeling model, and obtain a first labeling result output by the first type of labeling model, wherein the first type of labeling model is obtained by training based on the first sample data, and the data labels carried by the first sample data belong to the first label set;

[0136] The above-mentioned second data labeling unit 802 can be specifically used to input the data to be labeled into the second type of labeling model, label the data to be labeled by using the data labels contained in the second type of labeling model, and obtain the second labeling result output by the second type of labeling model, wherein the second type of labeling model is trained based on the second sample data, and the data labels carried by the second sample data belong to the second label set.

[0137] In one embodiment, the fusion unit 803 may be specifically used to perform consistency check on the first labeling result and the second labeling result. If the consistency check passes, a target labeling result matching the data to be labeled is determined from the first labeling result or the second labeling result.

[0138] In one embodiment, the first type of annotation model and the second type of annotation model may include a plurality of different data annotation models, respectively. The fusion unit 803 may be used to compare the first annotation results output by each of the first type of annotation models to obtain a first comparison result.

[0139] If the first comparison result indicates that the first annotation result is consistent and the first annotation result does not belong to the second tag set, it can be determined that the consistency check has passed, and the first annotation result is determined as the target annotation result that matches the data to be annotated;

[0140] If the first comparison result indicates that the first annotation result is consistent and the first annotation result belongs to the second label set, the second annotation results output by each second type of annotation model may be compared to obtain a second comparison result;

[0141] If the second comparison result indicates that the second annotation result is consistent, it can be determined that the consistency check has passed, and the second annotation result is determined as the target annotation result that matches the data to be annotated.

[0142] The first labeling results output by each of the first-type labeling models are compared to obtain the first comparison result, which may specifically include:

[0143] Respectively obtain first annotation results output by each first-category annotation model whose confidence is higher than a first confidence threshold, and compare the first annotation results whose confidence is higher than the first confidence threshold to obtain a first comparison result;

[0144] The second annotation results output by each second-type annotation model are compared to obtain the second comparison result, which may specifically include:

[0145] The second annotation results output by each second-category annotation model and having a confidence level higher than a second confidence threshold are respectively obtained, and the second annotation results having a confidence level higher than the second confidence threshold are compared to obtain a second comparison result.

[0146] The data labeling device described in the above embodiment can perform consistency verification on the labeling results output by multiple data labeling models constructed under different label sets, thereby ensuring that the target labeling results corresponding to the data to be labeled are obtained only when the labeling results are consistent, thereby further improving the accuracy and reliability of data labeling.

[0147] In one embodiment, Figure 8 The data annotation device shown may further include a correction annotation acquisition unit and a correction unit (not shown), wherein:

[0148] A revised annotation acquisition unit, used for acquiring a revised annotation result for the data to be annotated when the above consistency check fails;

[0149] The correction unit is used to correct the first type of annotation model and the second type of annotation model according to the data to be annotated and the correction annotation result, and trigger the above-mentioned first data annotation unit 801 to re-execute the step of annotating the data to be annotated by using the first type of annotation model to obtain the first annotation result, and trigger the second data annotation unit 802 to re-execute the step of annotating the data to be annotated by using the second type of annotation model to obtain the second annotation result, until the consistency check is passed.

[0150] The data labeling device described in the above embodiment can obtain the corresponding corrected labeling result when the target labeling result corresponding to the data to be labeled cannot be obtained, and feed it back to the above-mentioned data labeling models for correction to further improve the accuracy of data labeling.

[0151] See also Fig. 9 , Fig. 9 Schematic diagram of a modular electronic device disclosed in an embodiment of the present application. Fig. 9 As shown, the electronic device may include:

[0152] A memory 901 storing executable program codes;

[0153] A processor 902 coupled to the memory 901;

[0154] The processor 902 calls the executable program code stored in the memory 901 to execute all or part of the steps in any one of the data labeling methods described in the above embodiments.

[0155] In addition, an embodiment of the present application further discloses a computer-readable storage medium, which stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute all or part of the steps in any one of the data labeling methods described in the above embodiments.

[0156] In addition, an embodiment of the present application further discloses a computer program product. When the computer program product is run on a computer, the computer can execute all or part of the steps in any one of the data labeling methods described in the above embodiments.

[0157] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0158] The above is a detailed introduction to a data labeling method and device, an electronic device, and a storage medium disclosed in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for a person skilled in the art, according to the idea of ​​the present application, there may be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data labeling method, characterized in that: include: The data to be annotated is annotated by a first type of annotation model to obtain a first annotation result, wherein the first type of annotation model is constructed based on a first label set; the data annotation includes annotating the data to be annotated according to the type of data format, or annotating the data to be annotated according to the image recognition result, or annotating the data to be annotated according to the identifier corresponding to the data to be annotated; the type of data format includes at least one of an audio media format, an image media format, and a document format; the identifier corresponding to the data to be annotated includes at least one of a digital identifier, a character identifier, and a hash value identifier; The data to be labeled are labeled by a second type of labeling model to obtain a second labeling result, wherein the second type of labeling model is constructed based on a second label set, the first label set and the second label set are both subsets of a target label set corresponding to the data to be labeled, and the first label set is different from the second label set; and the second label set is a proper subset of the first label set; The first labeling result and the second labeling result are merged to obtain a target labeling result that matches the data to be labeled; Before annotating the data to be annotated by using the first type of annotation model to obtain a first annotation result, the method further includes: For each data tag included in the first tag set, respectively calculate the similarity between each data tag and other data tags in the first tag set to obtain similarities of multiple groups of data tag pairs; If the similarity of any set of data tag pairs meets the similarity condition, the data tags included in the data tag pairs whose similarity meets the similarity condition are added to the second tag set.

2. The method according to claim 1, characterized in that: Before annotating the data to be annotated by using the first type of annotation model to obtain a first annotation result, the method further includes: Determine a target label set corresponding to the data to be labeled according to the data labeling scenario corresponding to the data to be labeled; A first tag set is determined from the target tag set, where the first tag set includes a plurality of data tags belonging to the target tag set.

3. The method according to claim 1, characterized in that If the similarity of any set of data tag pairs meets the similarity condition, then adding the data tags included in the data tag pairs whose similarity meets the similarity condition to the second tag set includes: Sort the similarities of each group of data label pairs calculated from high to low similarities; The data tags included in the data tag pairs whose similarity sorting satisfies the order condition are added to the second tag set.

4. The method according to claim 1, characterized in that: Before calculating the similarity between each data tag included in the first tag set and other data tags in the first tag set to obtain similarities of multiple groups of data tag pairs, the method further includes: For each data label included in the first label set, respectively obtain the labeled data corresponding to each data label; According to the labeled data corresponding to each data label, obtain the label vector corresponding to each data label respectively; The step of calculating, for each data tag included in the first tag set, the similarity between each data tag and other data tags in the first tag set to obtain similarities of multiple groups of data tag pairs includes: The vector similarities between the label vector corresponding to each data label and the label vectors corresponding to other data labels in the first label set are calculated respectively to obtain the similarities of the corresponding multiple groups of data label pairs.

5. The method according to claim 4, characterized in that The step of obtaining a label vector corresponding to each data label according to the labeled data corresponding to each data label comprises: Mapping each first labeled data corresponding to the first data label to a first data vector corresponding to the first labeled data, wherein the first data label is any data label in the first label set; Average pooling is performed on each of the first data vectors to obtain a label vector corresponding to the first data label.

6. The method according to any one of claims 1 to 5, characterized in that: The step of annotating the data to be annotated by using the first type of annotation model to obtain a first annotation result includes: Inputting the data to be labeled into the first type of labeling model, labeling the data to be labeled using the data labels contained in the first type of labeling model, and obtaining a first labeling result output by the first type of labeling model, wherein the first type of labeling model is trained based on the first sample data, and the data labels carried by the first sample data belong to the first label set; The step of annotating the data to be annotated by using the second type of annotation model to obtain a second annotation result includes: The data to be labeled is input into the second type of labeling model, and the data to be labeled is labeled by using the data labels contained in the second type of labeling model to obtain a second labeling result output by the second type of labeling model, wherein the second type of labeling model is trained based on the second sample data, and the data labels carried by the second sample data belong to the second label set.

7. The method according to any one of claims 1 to 5, characterized in that: The step of fusing the first labeling result and the second labeling result to obtain a target labeling result that matches the data to be labeled includes: A consistency check is performed on the first labeling result and the second labeling result. If the consistency check passes, a target labeling result that matches the data to be labeled is determined from the first labeling result or the second labeling result.

8. The method according to claim 7, characterized in that The first type of annotation model includes a plurality of first annotation models, and the second type of annotation model includes a plurality of second annotation models, wherein the plurality of first annotation models correspond to different model architectures respectively, and the plurality of second annotation models correspond to different model architectures respectively; The performing a consistency check on the first labeling result and the second labeling result, and if the consistency check passes, determining a target labeling result that matches the to-be-labeled data from the first labeling result or the second labeling result, includes: Comparing the first annotation results output by each of the first annotation models to obtain a first comparison result; If the first comparison result indicates that the first annotation results output by each of the first annotation models are the same, and the first annotation results do not belong to the second label set, determining the first annotation results as target annotation results matched by the data to be annotated; If the first annotation result belongs to the second label set, comparing the second annotation results output by each of the second annotation models to obtain a second comparison result; If the second comparison result indicates that the second annotation results output by each of the second annotation models are the same, the second annotation result is determined as a target annotation result that matches the data to be annotated.

9. The method according to claim 8, characterized in that The comparing the first annotation results output by each of the first annotation models to obtain a first comparison result includes: Obtaining, from the first annotation results output by each of the first annotation models, first annotation results with a confidence level higher than a first confidence threshold, and comparing the first annotation results with a confidence level higher than the first confidence threshold to obtain a first comparison result; The step of comparing the second annotation results output by each of the second annotation models to obtain a second comparison result includes: From the second annotation results output by each of the second annotation models, second annotation results with confidence levels higher than a second confidence threshold are obtained, and the second annotation results with confidence levels higher than the second confidence threshold are compared to obtain a second comparison result.

10. The method according to claim 7, characterized in that After performing consistency check on the first annotation result and the second annotation result, the method further includes: If the consistency check fails, obtaining a corrected annotation result for the data to be annotated; According to the data to be labeled and the revised labeling result, the first type of labeling model and the second type of labeling model are respectively revised, and the data labeling of the data to be labeled by the first type of labeling model is re-executed to obtain the first labeling result, and the data labeling of the data to be labeled by the second type of labeling model is re-executed to obtain the second labeling result, until the consistency check is passed.

11. A data labeling device, characterized in that: include: A similarity calculation subunit, for calculating, for each data tag included in the first tag set, the similarity between each data tag and other data tags in the first tag set, to obtain similarities of multiple groups of data tag pairs; An adding subunit, configured to add the data tags included in the data tag pairs whose similarity satisfies the similarity condition to the second tag set if the similarity of any group of data tag pairs satisfies the similarity condition; A first data annotation unit is used to annotate the data to be annotated by using a first type of annotation model to obtain a first annotation result, wherein the first type of annotation model is constructed based on a first label set; the data annotation includes annotating the data to be annotated according to the type of data format, or annotating the data to be annotated according to the image recognition result, or annotating the data to be annotated according to the identifier corresponding to the data to be annotated; the type of data format includes at least one of an audio media format, an image media format, and a document format; the identifier corresponding to the data to be annotated includes at least one of a digital identifier, a character identifier, and a hash value identifier; A second data labeling unit is used to label the data to be labeled by using a second type of labeling model to obtain a second labeling result, wherein the second type of labeling model is constructed based on a second label set, the first label set and the second label set are both subsets of a target label set corresponding to the data to be labeled, and the first label set is different from the second label set; the second label set is a proper subset of the first label set; A fusion unit is used to fuse the first annotation result and the second annotation result to obtain a target annotation result that matches the data to be annotated.

12. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor implements the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images