A data labeling method and device, computer equipment and storage medium

Abstract

The disclosure provides a data labeling method and device, computer equipment and a storage medium, wherein the method comprises: filtering labeled objects from all initial objects by using first test data; sending the obtained to-be-labeled data and second test data to at least one labeled object according to a preset sending quantity ratio, and obtaining initial labeling results provided by each labeled object; in the case where the number of second test data labeled by the labeled object reaches a first preset number, determining the stage labeling error of the labeled object according to the standard labeling result corresponding to each second test data and the initial labeling result; for any to-be-labeled data, determining the target labeling result of the to-be-labeled data according to the stage labeling error and initial labeling result of each labeled object labeling the to-be-labeled data.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a data annotation method, apparatus, computer device, and storage medium. Background Technology

[0002] With the development of neural network technology, various machine learning models can be widely used. However, in most cases, before a machine learning model can be applied, iterative training with a large amount of labeled training data is required to ensure the prediction accuracy of the trained model.

[0003] However, conventional training data annotation methods require professional data annotators, which affects both the cost and efficiency of data annotation. Summary of the Invention

[0004] This disclosure provides at least one data annotation method, apparatus, computer device, and storage medium.

[0005] In a first aspect, embodiments of this disclosure provide a data annotation method, including:

[0006] Using the first test data, select labeled objects from the full set of initial objects;

[0007] According to a preset sending quantity ratio, the acquired data to be labeled and the second test data are sent to at least one of the labeled objects, and the initial labeling results provided by each labeled object are obtained;

[0008] When the number of second test data points annotated on the labeled object reaches a first preset number, the stage labeling error of the labeled object is determined based on the standard labeling result corresponding to each second test data point and the initial labeling result.

[0009] For any of the data to be labeled, the target labeling result of the data to be labeled is determined based on the stage labeling error and the initial labeling result of each of the labeling objects labeled with the data to be labeled.

[0010] In one possible implementation, determining the target annotation result of the data to be annotated based on the stage annotation error and initial annotation result of each of the annotation objects annotating the data to be annotated includes:

[0011] Based on the stage labeling error of each of the labeled objects labeled with the data to be labeled, target objects with stage labeling errors less than a preset error threshold are selected from the labeled objects;

[0012] When the number of target objects is greater than or equal to the second preset number, the fitting weight of each target object is determined according to the stage labeling error of each target object;

[0013] By using the fitting weights of each target object, the initial annotation results of each target object are fitted to obtain the target annotation results of the data to be annotated.

[0014] In one possible implementation, the step of using the first test data to filter out labeled objects from the full set of initial objects includes:

[0015] Based on the question-answering record information of the initial objects, determine the number of questions answered by each initial object within a preset historical time period;

[0016] Based on the number of questions answered, intermediate objects are selected from the initial objects;

[0017] Using the first test data, the labeled objects are selected from the intermediate objects.

[0018] In one possible implementation,

[0019] The step of using the first test data to filter out the labeled objects from the intermediate objects includes:

[0020] For any of the aforementioned intermediate objects, a third preset number of the first test data is sent to the intermediate object, and the test annotation results provided by the intermediate object for the first test data are obtained respectively.

[0021] Based on the standard annotation results and test annotation results of each of the first test data, determine the annotation accuracy of the intermediate object;

[0022] Based on the annotation accuracy, the annotated objects are selected from the intermediate objects.

[0023] In one possible implementation, after determining the stage annotation error of the labeled object, the method further includes:

[0024] If the stage annotation error of the labeled object is greater than or equal to a preset error threshold, the labeled object is removed; or,

[0025] If the stage annotation error of the labeled object is less than the preset error threshold, the number of the second test data annotated by the labeled object is updated to the default value, and the number of the second test data newly annotated by the labeled object is recounted according to the default value.

[0026] In one possible implementation, the data to be labeled includes a question to be answered and the answer content, and the target labeling result includes a content quality score used to characterize the quality of the answer content;

[0027] The data to be labeled is obtained according to the following steps:

[0028] Based on the display volume of each original data, a target number of intermediate data are selected from the original data;

[0029] Based on the answer content and annotation status of each intermediate data, data to be annotated is selected from the intermediate data; the annotation status is used to characterize whether the target annotation result of the intermediate data has been determined.

[0030] In one possible implementation, the step of filtering out data to be labeled from the intermediate data based on the answer content included in each of the intermediate data and the labeling status of the intermediate data includes:

[0031] If the number of characters in the answer content included in the intermediate data is less than the preset number of characters, it is determined whether the answer content contains sub-content with a preset format.

[0032] If not, determine whether the intermediate data is labeled data;

[0033] If not, then if the annotation status of the intermediate data indicates that the intermediate data has not yielded a target annotation result, the intermediate data shall be used as the data to be annotated.

[0034] In one possible implementation, after determining the target annotation result of the data to be annotated, the method further includes:

[0035] Using the data to be labeled and the target labeling results, the data labeling model is iteratively trained to obtain a trained data labeling model; the data labeling model is used to identify and process the input data to be labeled and output the predicted labeling results of the data to be labeled.

[0036] In one possible implementation, after determining the target annotation result of the data to be annotated, the method further includes:

[0037] Based on the content quality score represented by the target annotation results of each of the unannotated data, data to be recommended is selected from the unannotated data;

[0038] Based on the recommendation weight corresponding to the content quality score of each of the data to be recommended, the data to be recommended is pushed out.

[0039] Secondly, embodiments of this disclosure also provide a data annotation apparatus, comprising:

[0040] The filtering module is used to filter out labeled objects from the full set of initial objects using the first test data;

[0041] The sending module is used to send the acquired data to be labeled and the second test data to at least one of the labeled objects according to a preset sending quantity ratio, and to obtain the initial labeling results provided by each of the labeled objects;

[0042] The first determining module is used to determine the stage labeling error of the labeled object based on the standard labeling result corresponding to each of the second test data and the initial labeling result when the number of the second test data labeled on the labeled object reaches a first preset number.

[0043] The second determining module is used to determine the target labeling result of any of the labeled data based on the stage labeling error and initial labeling result of each of the labeled objects labeling the labeled data.

[0044] Thirdly, an optional implementation of this disclosure also provides a computer device, a processor, and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is configured to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the steps of the first aspect above, or any possible implementation of the first aspect, are performed.

[0045] Fourthly, an optional implementation of this disclosure also provides a computer-readable storage medium storing a computer program that, when run, performs the steps of the first aspect or any possible implementation of the first aspect.

[0046] For a description of the effects of the aforementioned data annotation device, computer equipment, and computer-readable storage medium, please refer to the description of the aforementioned data annotation method; it will not be repeated here.

[0047] The data annotation method, apparatus, computer equipment, and storage medium provided in this disclosure can filter all initial objects using first test data, identifying those suitable for data annotation. Since the initial objects may include objects that have not undergone data annotation, the filtered annotation objects may include non-professional annotation objects. Then, during the process of sending the data to be annotated to the annotation objects, by sending second test data with standard annotation results, the stage annotation error of the annotation objects can be determined using the second test data and its standard annotation results. Since the stage annotation error can be used to correct the annotation results, by finally using the initial annotation results of the annotation objects and the corresponding stage annotation errors, the target annotation result of the data to be annotated can be determined, ensuring that the annotation objects complete the annotation of the data to be annotated, and improving the accuracy of the determined target annotation result. Overall, the data annotation method provided in this disclosure not only enables accurate annotation of data to be annotated by non-professional annotation objects, increasing the number of objects that can be annotated and reducing annotation costs, but also improves the speed and efficiency of data annotation due to the increased number of objects.

[0048] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0050] Figure 1 A flowchart of a data annotation method provided by an embodiment of this disclosure is shown;

[0051] Figure 2 This illustration shows a schematic diagram of a specific implementation process of a data annotation method provided in an embodiment of this disclosure;

[0052] Figure 3 A schematic diagram of a data annotation apparatus provided in an embodiment of this disclosure is shown;

[0053] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown herein can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0055] Furthermore, the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein.

[0056] In this article, "multiple or several" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0057] Research has found that for content-based products (such as long articles, short articles, and Q&A topics), the quality of the content determines its usage. Therefore, improving content quality is crucial to increasing usage, for example, by retaining high-quality content and eliminating low-quality content. However, improving content quality requires accurately determining the content quality of each individual product, i.e., accurately labeling its content. Common content quality labeling methods include manual and machine labeling. Since machine labeling requires training with manually labeled sample data, manual labeling remains the fundamental approach. However, conventional manual labeling methods require highly trained professionals, making it difficult for most users to complete the task. This results in a limited number of labelers, making data labeling both costly and inefficient.

[0058] Based on the above research, this disclosure provides a data annotation method, apparatus, computer device, and storage medium. Using first test data, a full set of initial objects can be screened to identify those suitable for data annotation. Since the initial objects may include objects that have not undergone data annotation, the selected annotation objects may include non-professional annotation objects. Then, during the process of sending the data to be annotated to the annotation objects, by sending second test data with standard annotation results, the stage annotation error of the annotation objects can be determined using the second test data and its standard annotation results. Since the stage annotation error can be used to correct the annotation results, finally, by using the initial annotation results of the annotation objects and the corresponding stage annotation errors, the target annotation result of the data to be annotated can be determined, ensuring that the annotation objects complete the annotation of the data to be annotated, and improving the accuracy of the determined target annotation result. Overall, the data annotation method provided by the embodiments of this disclosure not only enables accurate annotation of data to be annotated by non-professional annotation objects, increases the number of objects that can be annotated, and reduces annotation costs, but also improves the speed and efficiency of data annotation due to the increased number of objects.

[0059] The shortcomings of the above solutions are the result of the inventor's practical experience and careful research. Therefore, the discovery process of the above problems and the solutions proposed in this disclosure below should be considered as the inventor's contribution to this disclosure.

[0060] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0061] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0062] To facilitate understanding of this embodiment, a data annotation method disclosed in this disclosure will first be described in detail. The execution subject of the data annotation method provided in this disclosure is generally a terminal device or other processing device with certain computing power. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, personal digital assistant device (PDA), handheld device, computer device, etc. In some possible implementations, the data annotation method can be implemented by the processor calling computer-readable instructions stored in the memory.

[0063] The data annotation method provided in this disclosure embodiment will be described below using a computer device as an example.

[0064] like Figure 1 The flowchart shown is a data annotation method provided in an embodiment of this disclosure, which may include the following steps:

[0065] S101: Using the first test data, select labeled objects from the full set of initial objects.

[0066] Here, the initial target audience can specifically be all users of content-based products, such as article reading applications, Q&A topic reading applications, and short video applications. For example, the initial target audience could be all users of an article reading application or all users of a short video application, and this initial target audience could include non-professional annotators.

[0067] For example, according to a preset annotation object confirmation period, all objects using content-based products within the current period can be identified, that is, the total number of initial objects corresponding to the current period can be determined. Here, the total number of initial objects may include objects that have not undergone data annotation operations and / or objects that have undergone data annotation operations.

[0068] The labeled object is the object selected from the entire initial object set, which can be used to annotate the data to be labeled. In specific implementation, the data to be labeled can be of any type. This embodiment of the disclosure takes data including unanswered questions and answer content as an example for illustration.

[0069] The first test data consists of data that has undergone multiple rounds of manual annotation, resulting in accurate annotations—standard annotation results—used to determine whether an object can be annotated. The data type of the first test data can be consistent with the data to be annotated. For example, if the data to be annotated is a question-and-answer topic including questions and answers, the first test data could be a test topic including sample questions and sample answers.

[0070] In practice, the first test data can be sent to each initial object in the full set of initial objects. Then, the test annotation results of each initial object after annotating the first test data can be obtained. For each initial object, the annotation accuracy can be determined using the standard annotation results and test annotation results corresponding to each set of first test data. If the annotation accuracy is greater than a preset accuracy, the initial object is selected for annotation; otherwise, if the annotation accuracy is less than or equal to the preset accuracy, the initial object is deemed unsuitable for annotation and is removed.

[0071] In one embodiment, the above-described S101 can be implemented according to the following steps:

[0072] S101-1: Based on the answer record information of the initial object, determine the number of questions answered by each initial object within the preset historical time period.

[0073] Here, the answer record information is information obtained by the initial object through authorization. It is used to indicate the information of each historical question answered by the initial object. The historical question information can specifically include the historical question, answer content, answer time, etc. The types of each question answered by the initial object are consistent with the types of questions to be answered in the data to be labeled.

[0074] The preset historical time period can be set based on experience, and this embodiment does not impose specific limitations. For example, the preset historical time period can be the past 30 days, the past 10 days, etc.

[0075] In practice, for any initial object, the number of questions answered by the initial object within a preset historical time period can be determined based on the initial object's answer record information, which is to say, the number of questions answered by the initial object is obtained.

[0076] S101-2: Based on the number of answers, select intermediate objects from the initial objects.

[0077] In practice, initial objects with more than a preset number of answers can be used as intermediate objects for filtering.

[0078] For example, with a preset historical time period of the past 30 days and a preset quantity of 30, intermediate objects that have answered more than 30 questions in the past 30 days can be selected from the full set of initial objects based on the number of questions answered by each initial object.

[0079] In this way, the more questions answered, the better it is to understand the data annotation process. Therefore, by using the number of questions answered and the preset number, suitable intermediate objects for data annotation can be selected from the full set of initial objects.

[0080] S101-3: Using the first test data, select labeled objects from the intermediate objects.

[0081] In practice, for each intermediate object, the first test data can be used to further test the intermediate object. If the intermediate object passes the test, it is used as a labeled object; otherwise, if the intermediate object fails the test, it is ignored.

[0082] By using the number of answers and the first test data to filter the entire initial set of objects twice, the rationality of the selected labeled objects can be improved.

[0083] In one embodiment, S101-3 described above can be implemented according to the following steps:

[0084] S101-3-1: For any intermediate object, send a third preset number of first test data to the intermediate object, and obtain the test annotation results provided by the intermediate object for the first test data respectively.

[0085] Here, the third preset number can be set based on experience, and this embodiment does not specify a specific setting. For example, the third preset number can be 30. The first test data can be a series of data to be labeled with standard annotation results. The test annotation result is the result of the intermediate object annotating the first test data.

[0086] In practice, a database for storing the first test data can be pre-established. For any intermediate object, a third preset number of first test data can be randomly selected from this database, and these selected first test data can be sent to the intermediate object. Then, the test annotation results after the intermediate object has annotated each of the first test data can be obtained.

[0087] S101-3-2: Determine the annotation accuracy of the intermediate object based on the standard annotation results and test annotation results of each first test data.

[0088] Here, the types and representation methods of the annotation results corresponding to the test data can be preset. For example, for any test data, the corresponding annotation result can be any value in the score range [0, 2]. The higher the score, the better the quality of the data content. For example, 0 points represents poor data quality, 1 point represents average data quality, and 2 points represents excellent data quality. Optionally, for any test data, the corresponding annotation result can be multiple pre-specified fixed values, such as 0, 1, and 2.

[0089] When the intermediate object annotates the first test data, the determined test annotation result is one of several pre-defined possible annotation results. The standard annotation result for a first test data point is also one of several pre-defined possible annotation results.

[0090] The labeling accuracy of each intermediate object can be determined by the ratio of the number of correctly labeled first test data to the third preset number.

[0091] In practice, for any intermediate object, it can be determined whether the test annotation results and standard annotation results of each first test data point labeled for that intermediate object are consistent. If they are consistent, the first test data point is determined to be correctly labeled; otherwise, it is determined to be incorrectly labeled. Based on this, the annotation accuracy of each intermediate object can be determined according to the standard annotation results and test annotation results of each first test data point.

[0092] S101-3-3: Select labeled objects from intermediate objects based on the labeling accuracy.

[0093] For example, for each intermediate object, if the annotation accuracy of the intermediate object is greater than the preset accuracy, the intermediate object can be used as an annotation object; otherwise, if the annotation accuracy of the intermediate object is less than or equal to the preset accuracy, the intermediate object can be determined to be an unlabelable object and removed.

[0094] S102: According to the preset sending quantity ratio, send the acquired data to be labeled and the second test data to at least one labeling object, and obtain the initial labeling results provided by each labeling object.

[0095] Here, the preset sending quantity ratio is used to indicate the ratio of the number of data to be labeled to the number of data sent to the second test data. For example, the preset quantity ratio can be 9:1, that is, after sending 9 pieces of data to be labeled to the labeling object, one piece of second test data will be sent to the labeling object. In specific implementation, the preset sending quantity ratio can be set according to experience, and this disclosure embodiment does not impose specific limitations.

[0096] The data to be labeled is the data that needs to be labeled. For example, the data to be labeled can be the content data of the previous N days, where N is greater than or equal to 1.

[0097] In one embodiment, the data to be labeled may include a question to be answered and the answer content, and the target labeling result may include a content quality score characterizing the quality of the answer content. The content quality score may fall within the score range [0, 2]. In specific implementation, the data to be labeled can be obtained according to the following steps:

[0098] Step 1: Based on the display volume of each original data, filter out the target number of intermediate data from the original data.

[0099] Here, raw data refers to data obtained from at least one content-based product. For example, data from an article reading app and a short video app. Impressions can represent the raw data's search volume, page views, etc., over the past N days within the content-based product. The target quantity can be a pre-specified number of data points. For example, the target quantity could be 100,000, 50,000, etc.

[0100] In practice, multiple raw data points from the previous day can be obtained from content-based products. Then, the raw data can be sorted according to their impressions, from highest to lowest, to obtain a sorting order for each piece of raw data. Afterward, the raw data points with a sorting order less than or equal to the target number can be used as intermediate data for filtering.

[0101] Step 2: Based on the answer content and annotation status of each intermediate data, filter out the data to be annotated from the intermediate data; the annotation status is used to indicate whether the target annotation result of the intermediate data has been determined.

[0102] For example, since content-based products are used by a large number of users every day, the intermediate data identified each day may contain duplicates. Some of this intermediate data may have already been labeled and has corresponding target annotation results, so there's no need for repeated labeling. Therefore, after obtaining the intermediate data, its labeling can be used to filter it, removing unlabeled raw data.

[0103] In practice, for each piece of intermediate data, it can be determined whether a target annotation result exists based on the annotation status of that intermediate data. If not, the answer content included in the intermediate data can be reviewed. If the review is passed, the intermediate data can be used as data to be annotated. If the review fails, the intermediate data can be ignored. The review of the answer content included in the intermediate data can specifically include content quality review, security review, sensitive word review, and content health review.

[0104] In one embodiment, step two above can be implemented according to the following steps:

[0105] S1: If the number of characters in the answer content included in the intermediate data is less than the preset number of characters, determine whether there is sub-content in the preset format in the answer content.

[0106] Here, the preset word count can be set based on experience, and this embodiment does not impose a specific limitation. For example, the preset word count can be 2000 words. The preset format can be an image / video format.

[0107] For example, the number of characters in the answer content included in the intermediate data can be determined, and it can be judged whether the number of characters in the answer is less than a preset number of characters. If the number of characters in the answer is less than the preset number of characters, it can be further judged whether there is sub-content with a preset format in the answer content. That is, it can be judged whether the answer content includes image content / video content. If there is sub-content with a preset format in the answer content, the intermediate data can be ignored.

[0108] S2: If not, determine whether the intermediate data is labeled data.

[0109] Here, labeled data can be data with initial labeling results but no target labeling results. As can be seen from the above embodiments, if the target object corresponding to a piece of data to be labeled is less than the second preset number, or if the stage labeling error of the labeling object corresponding to a piece of data to be labeled is less than the preset error threshold, the data to be labeled will be ignored. However, since this type of data to be labeled has already been distributed to the labeling objects for labeling, it is necessary to filter out the labeled data in order to prevent duplicate data sent to the labeling objects.

[0110] Therefore, in practice, if the intermediate data does not contain sub-content with a preset format, it can be further determined whether the intermediate data is already labeled. If so, the intermediate data can be ignored.

[0111] S3: If not, then if the annotation status of the intermediate data indicates that the target annotation result has not been determined, the intermediate data will be used as the data to be annotated.

[0112] For example, if the intermediate data does not belong to the labeled data, it can be determined whether the intermediate data has a target labeling result based on the labeling status of the intermediate data. If not, the intermediate data is regarded as data to be labeled; if the intermediate data has a target labeling result, the intermediate data can be ignored.

[0113] Based on the above S1 to S3, intermediate data with more than the preset number of words in the answer can be filtered out, intermediate data with sub-content in the preset format can be filtered out, labeled data can be filtered out, and intermediate data with target labeling results can be filtered out, so as to obtain the data to be labeled that is suitable for labeling the labeled object.

[0114] After obtaining all the data to be labeled, the candidate pool used to store the data can be updated using the latest identified data, resulting in an updated candidate pool. The candidate pool stores the latest data to be labeled, its data identifier, and the number of times it is displayed. For example, the candidate pool can be updated daily, such as at 3 PM.

[0115] The second test data can be manually labeled data or pre-labeled standard labeling results. It is used to determine the labeling error of the labeled object during the data labeling process. It can be the same data as the first test data or different data. The data type of the second test data can be consistent with the data to be labeled. For example, if the data to be labeled is a question-and-answer topic including questions to be answered and answer content, the second test data can be a test topic including sample questions to be answered and sample answer content.

[0116] The initial annotation results can be the annotation results provided by the annotation object for the data to be annotated and / or the annotation results provided by the annotation object for the second test data. A piece of data to be annotated can be distributed to a maximum of four preset number of annotation objects. For the same annotation object, a piece of data to be annotated will only be sent to that annotation object once.

[0117] In practice, for each piece of data to be labeled, it can be randomly distributed to no more than a fourth preset number of labeling objects. The initial labeling results provided by each labeling object for that piece of data can be obtained and saved. At the same time, the number of pieces of data to be labeled obtained by each labeling object can be recorded. If the number of pieces of data to be labeled obtained by any labeling object matches the number indicated by the preset sending quantity ratio, a second piece of test data can be sent to that labeling object, and the initial labeling results provided by that labeling object for the second test data can be obtained.

[0118] For example, for each labeled object, the data to be labeled and the second test data can be sent to the labeled object according to a preset sending quantity ratio. For instance, after sending 9 pieces of data to be labeled to the labeled object, one piece of second test data is sent to the labeled object interspersed. The process of sending data to be labeled to the labeled object can be divided into multiple stages, and in each stage, the quantity ratio between the data to be labeled and the second test data obtained by the labeled object conforms to the preset sending quantity ratio.

[0119] Regarding the storage of the initial annotation results provided by each annotation object, the data identifier of the data to be annotated, the object identifier of the annotation object, the initial annotation results provided by the annotation object, and the stage annotation error corresponding to the annotation object when providing the initial annotation results can be associated and stored in the database.

[0120] S103: When the number of second test data points for the labeled object reaches the first preset number, determine the stage labeling error of the labeled object based on the standard labeling results and initial labeling results corresponding to each second test data point.

[0121] Here, the stage annotation error is used to characterize the annotation error of the annotated object within a annotation stage. Specifically, this annotation error can be the Mean Absolute Error (MAE). The first preset number can be set empirically, and this embodiment does not impose a specific limitation. For example, the first preset number can be 20, 30, etc.

[0122] For example, for any labeled object, when the number of second test data labeled by the labeled object reaches a first preset number, the absolute error between the initial labeling result of the labeled object for each second test data and the standard labeling result corresponding to each second test data can be calculated respectively, and the MAE corresponding to the labeled object can be determined based on the absolute error corresponding to each second test data.

[0123] In one embodiment, the annotation of the second test data by the annotation object can be divided into multiple annotation stages. For each annotation stage, once the number of second test data annotated in that stage reaches a first preset quantity, a new annotation stage begins. For each annotation stage, the initial annotation result and the standard annotation result of the second test data annotated in that stage can be used to determine the stage annotation error corresponding to the annotation object in that stage. Furthermore, after determining the stage annotation error of the annotation object, the annotation object can be filtered using the stage annotation error to improve the accuracy of the finally determined target annotation result.

[0124] Specifically, after determining the stage annotation error of the labeled object, this stage annotation error can be compared with a preset error threshold. If the stage annotation error of the labeled object is greater than or equal to the preset error threshold, it indicates that the annotation error of the labeled object is large, the annotation accuracy in the corresponding annotation stage is low, and the accuracy of the initial annotation results of each data to be annotated cannot be guaranteed, thus having low reference value. To improve the accuracy of the final annotation results of each data to be annotated, labeled objects with stage annotation errors greater than or equal to the preset error threshold can be removed. In this way, labeled objects with large annotation errors can be deleted in a timely manner, ensuring the rationality and accuracy of the initial annotation results of each labeled object.

[0125] Conversely, if the stage annotation error of the labeled object is less than the preset error threshold, the number of the second test data labeled on the labeled object can be updated to the default value, and the number of the newly labeled second test data of the labeled object can be recounted according to the default value.

[0126] For example, if the stage annotation error is less than a preset error threshold, it indicates that the annotation error of the object is small, the initial annotation results of each data point to be annotated are accurate, and the reference value is high. Therefore, the object can be allowed to continue annotating new data. Thus, it can be determined that the object has entered a new annotation stage, and the number of second test data points annotated by the object is updated to a default value (specifically, 0). Afterwards, starting from 0, the number of newly annotated second test data points for the object in the new annotation stage can be recounted.

[0127] In one implementation, after identifying the annotation objects, a first preset number of second test data points can be sent to each annotation object. Based on the initial annotation results provided by each annotation object for the second test data and the standard annotation results of the second test data, the MAE (Maximum Annotation Equation) of each annotation object is determined. Annotation objects with an MAE greater than or equal to a preset error threshold are directly filtered out, while annotation objects with an MAE less than the preset error threshold are selected as the final filtered annotation objects, and the above-described S102 is executed.

[0128] In this way, by using the number of answers, the first test data, and the second test data, multiple screenings of the entire initial set of objects can be achieved, thereby improving the rationality of the selected labeled objects.

[0129] S104: For any data to be labeled, determine the target labeling result of the data based on the stage labeling error of each labeling object and the initial labeling result.

[0130] Here, the target annotation result can be the final annotation result determined from the data to be annotated.

[0131] For example, for any data to be labeled, the initial labeling results provided by each labeling object that has labeled the data can be obtained, along with the stage labeling error corresponding to each labeling object when providing the initial labeling results. Then, using the stage labeling error corresponding to each labeling object, the initial labeling results provided by each labeling object are corrected to obtain corrected labeling results. Finally, based on the corrected labeling results corresponding to each labeling object, the mean of the labeling results can be determined, and this mean can be used as the target labeling result for the data to be labeled.

[0132] In one embodiment, the above-described S104 can be implemented according to the following steps:

[0133] S104-1: Based on the stage annotation error of each annotation object in the data to be annotated, select target objects whose stage annotation error is less than the preset error threshold from the annotation objects.

[0134] Here, since the initial annotation results provided by each annotation object are saved, even after filtering some annotation objects using the stage annotation error corresponding to the annotation object, the initial annotation results of those objects will still be saved. Therefore, in determining the target annotation result, it is necessary to use the stage annotation error of each annotation object to further filter the saved initial annotation results to ensure that the final initial annotation results used are all highly accurate annotation results.

[0135] The preset error threshold can be set based on experience, and this embodiment of the disclosure does not impose specific limitations. For example, the preset error threshold can be 0.3, 0.4, etc.

[0136] In practice, the stage annotation error of each annotation object in the data to be annotated can be obtained from the database. Then, using the stage annotation error of each annotation object, target objects with stage annotation errors less than a preset error threshold can be selected from the annotation objects in the data to be annotated. Here, the selected target objects have smaller annotation errors, resulting in higher accuracy and reliability of the initial annotation results provided for the data to be annotated.

[0137] S104-2: When the number of target objects is greater than or equal to the second preset number, determine the fitting weight of each target object based on the stage annotation error of each target object.

[0138] Here, the second preset number can be the minimum number of target objects preset. If the number of target objects is less than the second preset number, it means that the number of initial annotation results that can be used to determine the target annotation result of the data to be annotated is small, which is not conducive to determining the accurate target annotation result. Therefore, the data to be annotated can be ignored and the target annotation result cannot be determined for the data to be annotated.

[0139] Conversely, if the number of target objects is greater than or equal to the second preset number, it means that there is a certain number of initial annotation results that can be used to determine the target annotation results of the data to be annotated. Then, according to the first mapping relationship between the pre-set annotation error range of each stage and the fitting weight, and the stage annotation error range to which the stage annotation error of each target object belongs, the fitting weight of each target object can be determined.

[0140] For example, the first mapping relationship can be: MAE∈(0.35, 0.4), with a fitting weight of 1; MAE∈(0.3, 0.35), with a fitting weight of 1.1; MAE∈(0, 0.3), with a fitting weight of 1.2.

[0141] S104-3: Using the fitting weights of each target object, the initial annotation results of each target object are fitted to obtain the target annotation results of the data to be annotated.

[0142] For example, after obtaining the fitting weights of each target object, the initial annotation results provided by each target object can be weighted and summed using the fitting weights of each target object to obtain the summation result. The summation result and the number of target objects are used to determine the mean of the summation, and the mean is used as the target annotation result of the data to be annotated.

[0143] In this way, the first test data can be used to filter all initial objects, identifying those suitable for data annotation. Since the initial objects may include objects that have not undergone data annotation, the filtered annotation objects may include non-professional annotation objects. Then, during the process of sending the data to be annotated to the annotation objects, by sending the second test data with standard annotation results, the stage annotation error of the annotation objects can be determined using the second test data and its standard annotation results. Since the stage annotation error can be used to correct the annotation results, by using the initial annotation results of the annotation objects and the corresponding stage annotation errors of the annotation objects, the target annotation result of the data to be annotated can be determined, ensuring that the annotation objects complete the annotation of the data to be annotated, and the accuracy of the determined target annotation result can be improved. Overall, the data annotation method provided by this embodiment can not only achieve accurate annotation of the data to be annotated by non-professional annotation objects, increase the number of objects that can be annotated, and reduce annotation costs, but also improve the speed and efficiency of data annotation due to the increased number of objects.

[0144] In one embodiment, after determining the target annotation result for the data to be labeled, the data to be labeled can also be used as training data to train the data labeling model, thereby improving the accuracy of the data labeling model. Specifically, the data labeling model can be iteratively trained using the data to be labeled and the target annotation result to obtain a trained data labeling model; the data labeling model is used to identify and process the input data to be labeled and output the predicted annotation result for the data to be labeled.

[0145] For example, the data to be labeled can be input into a data labeling model, which then performs identification processing on the data and outputs predicted labeling results. Next, the prediction loss can be determined using the target labeling results and the predicted labeling results of the data to be labeled, and the data labeling model can be iteratively trained using this prediction loss until a training cutoff condition is met, resulting in a well-trained data labeling model. The training cutoff condition can be reaching a preset number of iterations and / or achieving a preset prediction accuracy for the trained data labeling model.

[0146] In this way, by using a large amount of unlabeled data with determined target labeling results to train the data labeling model, the prediction accuracy of the data labeling model can be effectively improved.

[0147] Then, the trained data labeling model can be used to process the unlabeled data to be labeled, and output the predicted labeling results for each unlabeled data point. Based on the content quality score indicated by the predicted labeling results, high-quality data is selected from the unlabeled data to be labeled, and this high-quality data is then pushed to the system.

[0148] In another embodiment, after determining the target annotation results for the data to be annotated, recommendations can be made based on the target annotation results. Specifically, recommended data can be selected from the data to be annotated based on the content quality score represented by the target annotation results of each data point.

[0149] For example, based on the content quality score represented by the target annotation result of the data to be annotated, recommended data with a content quality score exceeding a first preset score can be filtered from the data to be annotated. Since a higher content quality score indicates better content quality of the data to be annotated, high-quality recommended data can be filtered from the data to be annotated using the content quality score and the first preset score.

[0150] Then, based on the recommendation weight corresponding to the content quality score of each piece of data to be recommended, the data to be recommended can be pushed.

[0151] Here, a second mapping relationship can be pre-defined between various quality score ranges and recommendation weights. Based on the content quality score of each piece of data to be recommended, the quality score range to which each piece of data belongs can be determined. Then, based on the second mapping relationship, the recommendation weight corresponding to each piece of data can be determined. Next, using the recommendation weights corresponding to each piece of data, the content quality score of each piece of data can be weighted to obtain a weighted score for each piece of data. Finally, the data to be recommended can be pushed in descending order of weighted score.

[0152] In this way, by filtering out the data to be recommended and pushing it according to a weighted score, high-quality content can be pushed, enabling users of content-based products to obtain high-quality data, which is conducive to improving the overall content quality of content-based products.

[0153] Optionally, after determining the target annotation results for the data to be annotated, data with content quality scores lower than a second preset score can be filtered out from the data to be annotated based on the content quality scores represented by the target annotation results of each data set. The second preset score is lower than the first preset score. In this way, by using the content quality score and the second preset score, lower-quality data to be updated can be filtered out from the data to be annotated. Afterwards, the data to be updated can be taken offline and updated to prevent the push of lower-quality data to the target audience.

[0154] In one embodiment, to facilitate understanding of the data annotation method provided in this disclosure, such as... Figure 2 The diagram shown illustrates a specific implementation process of a data annotation method provided in this embodiment, which may include the following steps:

[0155] S201: Intermediate data acquisition.

[0156] In practice, you can refer to step one above to filter out the target number of intermediate data. Furthermore, you can determine the first and second test data from the intermediate data.

[0157] S202: Determine the data to be labeled.

[0158] In practice, this step can be referred to as step two above.

[0159] S203: Update the candidate pool.

[0160] Here, the candidate pool can store the latest data to be labeled, the data identifier of the data to be labeled, the display volume of the data to be labeled, etc.

[0161] S204: Distribute the data to be labeled in the candidate pool to the labeling objects.

[0162] Regarding the above S204, it can be further subdivided into S205 to S209 as follows.

[0163] S205: Based on the number of answers, select intermediate objects from the full set of initial objects.

[0164] S206: Using the first test data, select labeled objects from the intermediate objects.

[0165] S207: Calculation of stage annotation error.

[0166] Specifically, the acquired data to be labeled and the second test data can be sent to at least one labeling object according to a preset sending quantity ratio. When the number of second test data labeled by the labeling object reaches a first preset quantity, the stage labeling error of the labeling object is determined based on the standard labeling result and the initial labeling result corresponding to each second test data.

[0167] S208: Object annotation permissions determined.

[0168] In practice, if the stage annotation error of the annotation object is greater than or equal to the preset error threshold, the annotation object can be removed; otherwise, the number of second test data annotated by the annotation object will be updated to the default value, and the number of newly annotated second test data of the annotation object will be recounted according to the default value.

[0169] S209: Obtain and store the initial annotation results provided by the annotation object.

[0170] Here, after sending the data to be labeled to the labeling object, the initial labeling results provided by the labeling object for the data to be labeled can be obtained and stored.

[0171] S210: Filter the data to be labeled and determine the target labeling results of the filtered data.

[0172] S210 can be further subdivided into S211 and S212 as follows.

[0173] S211: Stage labeling error screening.

[0174] In practice, this step can be referred to S104-1 above.

[0175] S212: Multi-object fitting is performed using a hierarchical weighted approach to obtain the target annotation results corresponding to the data to be annotated.

[0176] In practice, this step can refer to S104-2 and S104-3 above. Based on S212 above, the target annotation results corresponding to the data to be annotated can be obtained.

[0177] S213: Application of data to be labeled.

[0178] For example, the data annotation model can be trained using the unannotated data with target annotation results; or, the content quality score represented by the target annotation results of the unannotated data can be used to push the selected data to be recommended; or, the content quality score represented by the target annotation results of the unannotated data can be used to take the selected low-quality data offline and update it.

[0179] The specific implementation steps of S201 to S213 above can be referred to the above embodiments, and will not be repeated here.

[0180] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0181] Based on the same inventive concept, this disclosure also provides a data annotation device corresponding to the data annotation method. Since the principle of the device in this disclosure for solving the problem is similar to the data annotation method described above in this disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0182] like Figure 3 The diagram shown is a schematic representation of a data annotation device provided in an embodiment of this disclosure, comprising:

[0183] The filtering module 301 is used to filter out labeled objects from the full set of initial objects using the first test data;

[0184] The sending module 302 is used to send the acquired data to be labeled and the second test data to at least one of the labeled objects according to a preset sending quantity ratio, and to obtain the initial labeling results provided by each of the labeled objects;

[0185] The first determining module 303 is used to determine the stage labeling error of the labeled object based on the standard labeling result corresponding to each of the second test data and the initial labeling result when the number of the second test data labeled on the labeled object reaches a first preset number.

[0186] The second determining module 304 is used to determine the target labeling result of any of the labeled data based on the stage labeling error and initial labeling result of each of the labeled objects labeling the labeled data.

[0187] In one possible implementation, the second determining module 304, when determining the target annotation result of the data to be annotated based on the stage annotation error and initial annotation result of each of the annotation objects annotating the data to be annotated, is configured to:

[0188] Based on the stage labeling error of each of the labeled objects labeled with the data to be labeled, target objects with stage labeling errors less than a preset error threshold are selected from the labeled objects;

[0189] When the number of target objects is greater than or equal to the second preset number, the fitting weight of each target object is determined according to the stage labeling error of each target object;

[0190] By using the fitting weights of each target object, the initial annotation results of each target object are fitted to obtain the target annotation results of the data to be annotated.

[0191] In one possible implementation, the filtering module 301, when filtering labeled objects from the full set of initial objects using the first test data, is configured to:

[0192] Based on the question-answering record information of the initial objects, determine the number of questions answered by each initial object within a preset historical time period;

[0193] Based on the number of questions answered, intermediate objects are selected from the initial objects;

[0194] Using the first test data, the labeled objects are selected from the intermediate objects.

[0195] In one possible implementation, the filtering module 301, when filtering the labeled objects from the intermediate objects using the first test data, is configured to:

[0196] For any of the aforementioned intermediate objects, a third preset number of the first test data is sent to the intermediate object, and the test annotation results provided by the intermediate object for the first test data are obtained respectively.

[0197] Based on the standard annotation results and test annotation results of each of the first test data, determine the annotation accuracy of the intermediate object;

[0198] Based on the annotation accuracy, the annotated objects are selected from the intermediate objects.

[0199] In one possible implementation, the device further includes:

[0200] After determining the stage annotation error of the annotation object, the processing module 305 is further configured to:

[0201] If the stage annotation error of the labeled object is greater than or equal to a preset error threshold, the labeled object is removed; or,

[0202] If the stage annotation error of the labeled object is less than the preset error threshold, the number of the second test data annotated by the labeled object is updated to the default value, and the number of the second test data newly annotated by the labeled object is recounted according to the default value.

[0203] In one possible implementation, the data to be labeled includes a question to be answered and the answer content, and the target labeling result includes a content quality score used to characterize the quality of the answer content;

[0204] The device further includes:

[0205] The acquisition module 306 is used to acquire the data to be labeled according to the following steps:

[0206] Based on the display volume of each original data, a target number of intermediate data are selected from the original data;

[0207] Based on the answer content and annotation status of each intermediate data, data to be annotated is selected from the intermediate data; the annotation status is used to characterize whether the target annotation result of the intermediate data has been determined.

[0208] In one possible implementation, the acquisition module 306, when filtering out data to be labeled from the intermediate data based on the answer content included in each of the intermediate data and the labeling status of the intermediate data, is used to:

[0209] If the number of characters in the answer content included in the intermediate data is less than the preset number of characters, it is determined whether the answer content contains sub-content with a preset format.

[0210] If not, determine whether the intermediate data is labeled data;

[0211] If not, then if the annotation status of the intermediate data indicates that the intermediate data has not yielded a target annotation result, the intermediate data shall be used as the data to be annotated.

[0212] In one possible implementation, the device further includes:

[0213] Training module 307, after determining the target annotation result of the data to be annotated, is used to:

[0214] Using the data to be labeled and the target labeling results, the data labeling model is iteratively trained to obtain a trained data labeling model; the data labeling model is used to identify and process the input data to be labeled and output the predicted labeling results of the data to be labeled.

[0215] In one possible implementation, the device further includes:

[0216] Recommendation module 308, after determining the target annotation result of the data to be annotated, is used for:

[0217] Based on the content quality score represented by the target annotation results of each of the unannotated data, data to be recommended is selected from the unannotated data;

[0218] Based on the recommendation weight corresponding to the content quality score of each of the data to be recommended, the data to be recommended is pushed out.

[0219] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0220] Based on the same technical concept, embodiments of this application also provide a computer device. (Refer to...) Figure 4 The diagram shown is a structural schematic of a computer device provided in an embodiment of this application, comprising:

[0221] The system includes a processor 41, a memory 42, and a bus 43. The memory 42 stores machine-readable instructions executable by the processor 41. The processor 41 executes these machine-readable instructions, performing the following steps: S101: Using first test data, select labeled objects from the full set of initial objects; S102: According to a preset sending ratio, send the acquired data to be labeled and second test data to at least one labeled object, and obtain the initial labeling results provided by each labeled object; S103: When the number of second test data labeled by a labeled object reaches a first preset number, determine the stage labeling error of the labeled object based on the standard labeling results and initial labeling results corresponding to each second test data; and S104: For any data to be labeled, determine the target labeling result of the data to be labeled based on the stage labeling errors and initial labeling results of each labeled object labeling the data to be labeled.

[0222] The aforementioned memory 42 includes a main memory 421 and an external memory 422. The main memory 421, also known as internal memory, is used to temporarily store the computational data in the processor 41, as well as the data exchanged with external memory such as a hard disk. The processor 41 exchanges data with the external memory 422 through the main memory 421. When the computer device is running, the processor 41 and the memory 42 communicate through the bus 43, so that the processor 41 executes the execution instructions mentioned in the above method embodiments.

[0223] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the data annotation method described in the above-described method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0224] The computer program product of the data annotation method provided in this disclosure includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the steps of the data annotation method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0225] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0226] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0227] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0228] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0229] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0230] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the user through pop-up information or by asking the user to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0231] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A data annotation method, characterized in that, include: Using the first test data, labeled objects are selected from the full set of initial objects. The first test data is data of content-type products with standard labeling results. The first test data is used to determine whether the initial objects can be used to label the content quality of the content-type products. According to a preset sending quantity ratio, the acquired data to be labeled and the second test data are sent to at least one of the labeled objects, and the initial labeling results provided by each labeled object are obtained. The second test data is the data of the content product with standard labeling results. The second test data is used to determine the labeling error of the labeled object in the data labeling process. When the number of second test data points annotated on the labeled object reaches a first preset number, the stage labeling error of the labeled object is determined based on the standard labeling result corresponding to each second test data point and the initial labeling result, wherein the stage labeling error is used to correct the initial labeling result; For any of the data to be labeled, the target labeling result of the data to be labeled is determined based on the stage labeling error and initial labeling result of each of the labeling objects labeled with the data to be labeled. The data to be labeled with the target labeling result is used to train the data labeling model of the content product, or the target labeling result is used to recommend the data to be labeled in the content product.

2. The method according to claim 1, characterized in that, The step of determining the target annotation result of the data to be annotated based on the stage annotation error and initial annotation result of each of the annotation objects annotating the data to be annotated includes: Based on the stage labeling error of each of the labeled objects labeled with the data to be labeled, target objects with stage labeling errors less than a preset error threshold are selected from the labeled objects; When the number of target objects is greater than or equal to the second preset number, the fitting weight of each target object is determined based on the stage labeling error of each target object; By using the fitting weights of each target object, the initial annotation results of each target object are fitted to obtain the target annotation results of the data to be annotated.

3. The method according to claim 1, characterized in that, The step of using the first test data to filter out labeled objects from the full set of initial objects includes: Based on the question-answering record information of the initial objects, determine the number of questions answered by each initial object within a preset historical time period; Based on the number of questions answered, intermediate objects are selected from the initial objects; Using the first test data, the labeled objects are selected from the intermediate objects.

4. The method according to claim 3, characterized in that, The step of using the first test data to filter out the labeled objects from the intermediate objects includes: For any of the aforementioned intermediate objects, a third preset number of the first test data is sent to the intermediate object, and the test annotation results provided by the intermediate object for the first test data are obtained respectively. Based on the standard annotation results and test annotation results of each of the first test data, determine the annotation accuracy of the intermediate object; Based on the annotation accuracy, the annotated objects are selected from the intermediate objects.

5. The method according to claim 1, characterized in that, After determining the stage annotation error of the annotation object, the method further includes: If the stage annotation error of the labeled object is greater than or equal to a preset error threshold, the labeled object is removed; or, If the stage annotation error of the labeled object is less than the preset error threshold, the number of the second test data annotated by the labeled object is updated to the default value, and the number of the second test data newly annotated by the labeled object is recounted according to the default value.

6. The method according to claim 1, characterized in that, The data to be labeled includes questions to be answered and answer content, and the target labeling result includes a content quality score used to characterize the quality of the answer content; The data to be labeled is obtained according to the following steps: Based on the display volume of each original data, a target number of intermediate data are selected from the original data; Based on the answer content and annotation status of each intermediate data, data to be annotated is selected from the intermediate data; the annotation status is used to characterize whether the target annotation result of the intermediate data has been determined.

7. The method according to claim 6, characterized in that, The step of filtering out data to be labeled from the intermediate data based on the answer content and labeling status of each intermediate data includes: If the number of characters in the answer content included in the intermediate data is less than the preset number of characters, it is determined whether the answer content contains sub-content with a preset format. If not, determine whether the intermediate data is labeled data; If not, then if the annotation status of the intermediate data indicates that the intermediate data has not yielded a target annotation result, the intermediate data shall be used as the data to be annotated.

8. The method according to claim 1, characterized in that, After determining the target annotation result for the data to be annotated, the process also includes: Using the data to be labeled and the target labeling results, the data labeling model is iteratively trained to obtain a trained data labeling model; the data labeling model is used to identify and process the input data to be labeled and output the predicted labeling results of the data to be labeled.

9. The method according to claim 1, characterized in that, After determining the target annotation result for the data to be annotated, the process also includes: Based on the content quality score represented by the target annotation results of each of the unannotated data, data to be recommended is selected from the unannotated data; Based on the recommendation weight corresponding to the content quality score of each of the data to be recommended, the data to be recommended is pushed out.

10. A data annotation device, characterized in that, include: The filtering module is used to filter out labeled objects from the full set of initial objects using the first test data, wherein the first test data is data of content-type products with standard labeling results, and the first test data is used to determine whether the initial objects can be used to label the content quality of the content-type products. The sending module is used to send the acquired data to be labeled and the second test data to at least one of the labeled objects according to a preset sending quantity ratio, and to obtain the initial labeling results provided by each of the labeled objects. The second test data is the data of the content product with standard labeling results. The second test data is used to determine the labeling error of the labeled object in the data labeling process. The first determining module is used to determine the stage labeling error of the labeled object based on the standard labeling result corresponding to each of the second test data and the initial labeling result when the number of the second test data labeled on the labeled object reaches a first preset number, wherein the stage labeling error is used to correct the initial labeling result; The second determining module is used to determine the target labeling result of any of the data to be labeled based on the stage labeling error and initial labeling result of each of the labeling objects labeling the data to be labeled. The data to be labeled with the target labeling result is used to train the data labeling model of the content product, or the target labeling result is used to recommend the data to be labeled in the content product.

11. A computer device, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, the processor executing the machine-readable instructions stored in the memory, wherein when the machine-readable instructions are executed by the processor, the processor performs the steps of the data annotation method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer device, performs the steps of the data annotation method as described in any one of claims 1 to 9.

Images