Training method of data annotation model, data annotation method and corresponding device
By disassembling the annotation rules of the annotation task and using thinking chain information to train the data annotation model, the problem of manual annotation dependence and inconsistent labeling in the existing technology is solved, and efficient and accurate automatic data annotation is achieved.
Patent Information
- Application Number
- CN202311315584.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-11
- Publication Date
- 2025-04-18
AI Technical Summary
Existing data annotation methods rely on manual annotation, which is affected by the business level of the marker, and the labeling categories are inconsistent, with low accuracy and efficiency.
By obtaining sample data that meets the preset annotation conditions, the annotation rules for the annotation task are disassembled into a single annotation condition, and the model is trained using thinking chain information to obtain the target data annotation model, which is used to automatically perform data annotation.
It improves the training efficiency of the data annotation model, reduces the dependence of manual annotation, ensures the consistency of the annotation categories, improves the accuracy and efficiency of the annotation, and provides the target thinking chain information corresponding to the category information to understand the annotation reasons.
Smart Images
Figure CN119851271A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing technology, and in particular, to a training method for a data annotation model, a data annotation method, and a corresponding device. Background Art
[0002] In the field of data labeling, data is usually manually labeled based on the labeling rules in the labeling task. However, the professional levels of different labelers vary. Especially for labeling tasks in professional fields, labelers are required to have knowledge of the professional field. Affected by the professional levels of different labelers, different labelers may have different labeling categories for the same data, and the accuracy of the labeling categories cannot be guaranteed. In addition, the efficiency of manual labeling is low. Summary of the invention
[0003] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.
[0004] In a first aspect, the present disclosure provides a method for training a data annotation model, the method comprising:
[0005] Acquire sample data that meets a preset labeling condition, wherein the preset labeling condition includes a single labeling condition obtained by decomposing a labeling rule corresponding to a labeling task, and the sample data corresponds to thought chain information, wherein the thought chain information is used to indicate whether the sample data belongs to a labeling category corresponding to the labeling task and the corresponding reason;
[0006] Determine a sample question corresponding to the sample data according to the marked category and the sample data, and determine a sample answer corresponding to the sample data according to the thought chain information;
[0007] Model training is performed based on the sample questions and the sample answers to obtain a target data labeling model. The target data labeling model is used to obtain answers to input questions including target data, wherein the answers include category information of the target data and target thinking chain information corresponding to the category information.
[0008] In a second aspect, the present disclosure provides a data annotation method, the method comprising:
[0009] Get questions that include target data;
[0010] Inputting the question into a target data annotation model to obtain an answer to the question, wherein the answer includes category information of the target data and target thought chain information corresponding to the category information, wherein the target data annotation model is obtained by model training according to the data annotation model training method described in any one of the first aspects;
[0011] The target data is labeled according to the category information.
[0012] In a third aspect, the present disclosure provides a training device for a data annotation model, the device comprising:
[0013] A first acquisition module is used to acquire sample data that meets a preset labeling condition, wherein the preset labeling condition includes a single labeling condition obtained by decomposing a labeling rule corresponding to a labeling task, and the sample data corresponds to thought chain information, wherein the thought chain information is used to indicate whether the sample data belongs to a labeling category corresponding to the labeling task and the corresponding reason;
[0014] A first determination module is used to determine a sample question corresponding to the sample data according to the marked category and the sample data, and to determine a sample answer corresponding to the sample data according to the thought chain information;
[0015] A training module is used to perform model training based on the sample questions and the sample answers to obtain a target data labeling model. The target data labeling model is used to obtain answers to input questions including target data, and the answers include category information of the target data and target thinking chain information corresponding to the category information.
[0016] In a fourth aspect, the present disclosure provides a data labeling device, the device comprising:
[0017] A second acquisition module, used for acquiring questions including target data;
[0018] an input module, used to input the question into a target data annotation model to obtain an answer to the question, wherein the answer includes category information of the target data and target thought chain information corresponding to the category information, wherein the target data annotation model is obtained by model training according to the training device for the data annotation model described in the third aspect;
[0019] A labeling module is used to label the target data according to the category information.
[0020] In a fifth aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in any one of the first aspect or the second aspect.
[0021] In a sixth aspect, the present disclosure provides an electronic device, including:
[0022] a storage device having a computer program stored thereon;
[0023] A processing device is used to execute the computer program in the storage device to implement the steps of the method in any one of the first aspect or the second aspect.
[0024] Through the above technical solution, the labeling rules corresponding to the labeling task are decomposed into a single labeling condition, which can facilitate the model to understand and learn the labeling process and improve the efficiency of model training. The subsequent data labeling through the data labeling model is not affected by the business level of the labeler, and there is no problem of inconsistent labeling categories for the same data. The accuracy and efficiency are higher than manual labeling. In addition, through the target thinking chain information corresponding to the category information, users can clearly understand the reasons for determining the category information.
[0025] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings:
[0027] Figure 1 is a manually annotated flow chart provided according to an exemplary embodiment;
[0028] Figure 2 is a flowchart of a method for training a data annotation model provided according to an exemplary embodiment;
[0029] Figure 3 is a schematic diagram of decomposing a labeling task according to an exemplary embodiment;
[0030] Figure 4 is a schematic diagram of sample data provided according to an exemplary embodiment;
[0031] Figure 5 is a schematic diagram of sample data provided according to an exemplary embodiment;
[0032] Figure 6 is a schematic diagram of sample data provided according to an exemplary embodiment;
[0033] Figure 7 is a training flow chart of a data annotation model provided according to an exemplary embodiment;
[0034] Figure 8 is a flowchart of a data labeling method provided according to an exemplary embodiment;
[0035] Fig. 9 is a block diagram of a training device for a data annotation model provided according to an exemplary embodiment;
[0036] Fig.10 is a block diagram of a data labeling device provided according to an exemplary embodiment;
[0037] Fig.11 is a block diagram of an electronic device provided according to an exemplary embodiment. DETAILED DESCRIPTION
[0038] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0039] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0040] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0041] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0042] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0043] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0044] AIGC (Artificial Intelligence Generated Content) technology can complete various data tasks, such as automatic annotation, automatic writing, machine translation, etc., by automatically generating natural language, images, videos, and voice content. In the field of data annotation, data is usually annotated manually based on the annotation rules in the annotation task, such as the Standard Operation Procedure (SOP). Or the data is preliminarily screened through the model, and then manually reviewed and annotated, which means that manual intervention by annotators is required.
[0045] Since the professional levels of different labelers vary, especially for labeling tasks in professional fields, labelers are required to have knowledge of the professional field. Depending on whether labeling requires knowledge of professional fields, labelers can be divided into labelers based on professional knowledge and labelers based on common sense. Labelers based on professional knowledge usually need to master knowledge in a specific field to complete specific labeling tasks. For example, labelers in advertising systems need to master knowledge related to advertising laws and advertising placement rules, which require long-term training in the early stage to master, and once certain rules change, they need to be retrained. This type of labeler has a long training cycle and cannot respond to changes quickly and handle tasks immediately. Common sense labelers usually do not need to master knowledge in a specific field. They only need to label data based on the steps and rules defined in the SOP to complete the labeling task. This type of labeler has a short training cycle, can get started quickly, and has a low threshold.
[0046] Generally speaking, data labeling tasks are completed by the above two types of labelers. Figure 1 As shown in the figure, annotators based on professional knowledge learn the labeling operation process of the labeling task through their own professional knowledge, and then label the data to be labeled, and finally output the category information or other information to which the data belongs. However, even annotators based on professional knowledge will encounter some problems when labeling data.
[0047] For example, if three labelers label a piece of data, it is possible that the three labelers label it in different categories, so it is difficult to know the true category of the data. This situation usually occurs in scenarios with complex business, complex rules, or involving a wide range of knowledge, which means that the consistency rate of manual labeling is low.
[0048] For example, each labeler has a different subjective understanding of the SOP, so no matter how the training process is aligned, there will always be deviations. In particular, the deviations will be greater for complex rules, making it difficult to ensure the accuracy of manual labeling. For example, out of 100 data, a labeler may only be able to label 80 correctly, which is a low accuracy rate.
[0049] It is also obvious that manual labeling is less efficient than automatic labeling with models, and as the amount of labeling increases and the number of labelers increases, the labeling discrepancy rate will also increase. In addition, manual labeling has greater language restrictions. For example, there are fewer labelers for minority languages, so it is difficult to manually label data in minority languages.
[0050] In view of this, the present disclosure provides a data annotation model training method, a data annotation method and a corresponding device to solve the above technical problems.
[0051] The embodiments of the present disclosure are further explained and illustrated below in conjunction with the accompanying drawings.
[0052] Figure 2 is a flowchart of a method for training a data annotation model according to an exemplary embodiment of the present disclosure, with reference to Figure 2 , the method may include the following steps:
[0053] S201: Obtain sample data that meets preset labeling conditions.
[0054] Among them, the preset labeling conditions include a single labeling condition obtained by disassembling the labeling rules corresponding to the labeling task, and the sample data corresponds to the thinking chain information, which is used to indicate whether the sample data belongs to the labeling category corresponding to the labeling task and the corresponding reasons.
[0055] S202. Determine a sample question corresponding to the sample data based on the labeled category and the sample data, and determine a sample answer corresponding to the sample data based on the thinking chain information.
[0056] S203: Perform model training based on sample questions and sample answers to obtain a target data labeling model.
[0057] Among them, the target data annotation model is used to obtain the answer to the question based on the input question including the target data, and the answer includes the category information of the target data and the target thinking chain information corresponding to the category information.
[0058] By using the above method, the labeling rules corresponding to the labeling task are decomposed into a single labeling condition, which can facilitate the model to understand and learn the labeling process and improve the efficiency of model training. The subsequent data labeling through the data labeling model is not affected by the professional level of the labeler, and there is no problem of inconsistent labeling categories for the same data. The accuracy and efficiency are higher than manual labeling. In addition, through the target thinking chain information corresponding to the category information, users can clearly understand the reasons for determining the category information.
[0059] In a possible manner, obtaining sample data that meets preset labeling conditions may include: obtaining sample data in at least one of the following ways: obtaining first sample data that meets each single labeling condition and does not meet the exemption condition; obtaining second sample data that simultaneously meets multiple single labeling conditions and does not meet the exemption condition; obtaining third sample data that meets at least one single labeling condition and meets the exemption condition, wherein the exemption condition represents the condition for not labeling the sample data with a labeling category.
[0060] It should be understood that there are usually standard operating procedures (SOP) documents for labeling tasks, and the training and labeling of labelers are completed based on these SOP documents. In this process, for different rules, labelers need to remember a large number of unfamiliar concepts, which often puts a heavy burden on labelers. Therefore, for specific rules and categories, the rules or categories can be split until the split rules or categories can be understood based on common sense, and this splitting process can be continuously optimized according to needs.
[0061] For example, the split rules may include the following three situations:
[0062] 1. A situation that meets a certain rule appears in the data to be labeled;
[0063] 2. The data to be labeled meets multiple rules at the same time;
[0064] 3. The data to be labeled meets one or more rules, but is an exception or exemption.
[0065] Furthermore, corresponding sample data may be obtained for at least one of the above three situations to cover situations that may occur in actual business scenarios and improve the accuracy of the data annotation model.
[0066] In possible ways, the single labeling condition is obtained by: obtaining the standard operating procedure document corresponding to the labeling task; disassembling the labeling rules for the labeling category in the standard operating procedure document to obtain the single labeling condition corresponding to the associated category of the labeling category; or, disassembling the labeling rules for the labeling category in the standard operating procedure document to obtain the single labeling condition corresponding to the subcategory of the labeling category, and determining the single labeling condition corresponding to the associated category of the subcategory.
[0067] For example, refer to Figure 3 All labeling tasks can be divided into different labeling categories or rules, each rule (Category) can be divided into common-sense labeling conditions or provisions (provisions), and the difficulty of the labeling conditions can also be divided according to the above three situations, such as satisfying a single labeling condition is a simple labeling condition, satisfying multiple single labeling conditions at the same time is a medium labeling condition, satisfying at least one single labeling condition and satisfying the exemption rule is a difficult labeling condition, and so on. The present disclosure does not limit this.
[0068] It is worth mentioning that by making the SOP of the labeling task common sense, the complex SOP is broken down into labeling conditions that are easier for the model to learn and understand, thereby helping the model learn the labeling process. For example, a certain labeling task requires labeling whether an image belongs to the category of smoking or cigarettes. The content of the SOP is as follows: If the image contains visual features that encourage or induce smoking or any cigarette-related features, the image can be labeled with that category; if the image context is a flavor product, it does not belong to that category. In the above SOP, if the labeler is asked to perform manual labeling, since no specific information about cigarette products and categories is provided, and no specific information about flavor products is provided, the labeler needs to learn and master the relevant knowledge, which takes a lot of time.
[0069] For example, the content of the above SOP can be broken down into common sense content, such as:
[0070] 1. If the following content appears in the image, it can be marked with this category: cigarettes, cigars, chewing tobacco, pipes, hookahs, hookah lounges, rolling papers, vaporization delivery devices, and electronic cigarettes;
[0071] 2. If there is tobacco-related text in the image, it can be marked with that category;
[0072] 3. If there is any smoking-related behavior in the image, it can be marked with this category;
[0073] 4. If the image contains text or symbols prohibiting smoking or opposing smoking, this category will not be marked;
[0074] 5. If flavor products appear in the image context, but there are also tobacco-related references, this category will not be marked;
[0075] 6. If the image is a screenshot or footage from a movie, television program, video game, or other media, it will not be labeled with this category even if visual features related to smoking appear.
[0076] That is to say, the marked category can be disassembled to obtain the associated category corresponding to the marked category, for example, smoking-related behaviors, tobacco-related text content, etc. belong to the associated category of the smoking or cigarette category, and then determine the single marking condition corresponding to the associated category. The marked category can also be disassembled to obtain the subcategory corresponding to the marked category, for example, cigarettes, cigars, chewing tobacco, pipes, hookahs, electronic cigarettes, etc. are all subcategories under the smoking or cigarette category, and then obtain a single marking condition for the subcategory. It is also possible to determine the single marking condition corresponding to the associated category of the subcategory, such as hookah lounge is the associated category corresponding to hookah, and so on. The specific disassembly can be carried out according to the actual situation, and the present disclosure does not limit this.
[0077] In a possible manner, the exemption conditions include at least one of the following conditions: the sample data contains content that indicates that the sample data does not belong to the marked category; the sample data contains content of other categories besides the marked category; the data source of the sample data meets the preset conditions.
[0078] For example, we continue to refer to the content of the above SOP and disassemble it to obtain the content that belongs to common sense. We can also disassemble the exemption rules in the SOP to obtain the exemption conditions. Among them, there are contents in the sample data that represent that the sample data does not belong to the marked category, such as fruit-related pictures, which do not belong to the smoking or cigarette category. There are contents of other categories in the sample data besides the marked category, such as no-smoking texts and symbols, flavor products containing tobacco, etc. The data source of the sample data meets the preset conditions, such as screenshots or shots of movies, TV shows, video games or other media. The specific conditions can be determined according to the needs, and the present disclosure does not limit this.
[0079] It can be found that by disassembling the SOP, the labeling tasks that require professional knowledge are transformed into common sense labeling tasks, and the abstract definitions can be transformed into specific rules, making model construction simpler and greatly improving the model performance. And after converting professional definitions into common sense rules, even non-professional ordinary labelers or models with common sense knowledge can complete the labeling tasks. In this way, professional labelers can perform the common sense disassembly of SOP, without consuming a large number of professional labelers for manual labeling, which improves the labeling efficiency.
[0080] In a possible manner, obtaining sample data that meets preset labeling conditions may include: generating sample data according to the preset labeling conditions and a data generation model; and / or searching for sample data in a sample database according to the preset labeling conditions.
[0081] For example, sample data can be generated according to preset annotation conditions and data generation models. The data generation model can be a text-to-text model, a text-to-image model, etc., which can be selected according to specific needs. For example, a text2image model can be selected to generate sample data in the form of images, and the present disclosure does not limit this. Sample data can also be searched in a search engine or a sample library according to preset annotation conditions, and the present disclosure does not limit this.
[0082] For example, we will take the above three situations as an example to obtain sample data corresponding to the above three situations. Figure 4 The image shown contains only the visual features of cigars. Get sample data that satisfies multiple single annotation conditions at the same time, such as Figure 5 The image shown contains a cigar and shows smoking behavior. Get sample data that meets at least one single annotation condition and meets the exemption rules, such as Figure 6 The image shown contains a cigar but is a no smoking sign. A certain amount of data can be obtained according to the needs, for example, 100 images are obtained for each case, and the present disclosure does not limit this. Images that do not meet the preset annotation conditions can also be obtained, so as to cover the situations that may occur in actual business scenarios and improve the accuracy of the data annotation model.
[0083] In possible ways, the thinking chain information corresponding to the sample data is determined by at least one of the following ways: obtaining the thinking chain information manually configured for the sample data; when the preset labeling conditions include multiple labeling conditions of different types, determining the target labeling conditions satisfied by the sample data in the preset labeling conditions, and determining the thinking chain information corresponding to the sample data based on the target labeling conditions and the preset thinking chain template; obtaining the initial thinking chain information corresponding to the sample data through a model for constructing the thinking chain information, and obtaining the manual verification results of the initial thinking chain information, and determining the thinking chain information corresponding to the sample data based on the manual verification results and the initial thinking chain information.
[0084] For example, for the sample data corresponding to each preset annotation condition, a specific chain of thought (Chain of Thought, CoT) is constructed to help the model better learn the annotation process. Taking the sample data corresponding to the above three situations as examples, for example, the chain of thought constructed for the picture in situation 1 can be "Yes. Since the image contains cigars, there is a tendency to induce smoking, so the image belongs to the smoking / cigarette category", the chain of thought constructed for the picture in situation 2 can be "Yes. Since the image contains cigars, there is smoking behavior, and there is a tendency to induce smoking, the image belongs to the smoking / cigarette category", and the chain of thought constructed for the picture in situation 3 can be "No. There are cigarettes in the image, but since the image is to indicate that smoking is prohibited, the image does not belong to the smoking / cigarette category".
[0085] For example, it can be automatically generated according to preset labeling conditions and preset thought chain templates. For example, the preset thought chain template can be "Yes. Because the image contains XX, there is a tendency to induce smoking, so the image belongs to the smoking / cigarette category." For the image containing cigars obtained in situation 1, "cigar" can be filled in the "XX" in the template to obtain the thought chain information. In addition, the above-mentioned thought chain information can also be manually configured, and a model of the thought chain information can be constructed to generate the thought chain information, and then the final thought chain information can be manually reviewed and determined, etc. The present disclosure does not limit this. In this way, the thought chain information of the sample data can be obtained, which is convenient for subsequent model output to help users understand the reasons for determining the labeling category.
[0086] Further, after determining the thought chain information of the sample data, a question-answer pair (QA pair) of the sample data may be constructed.
[0087] For example, Figure 3 As shown in the figure, for the image labeling task, the global prompt can be "Analyze the following image", the description part is the description of the problem, for example, it can be "Determine whether the image belongs to the smoking / cigarette category", and finally add the sample data to obtain the sample problem for the sample image, such as "Analyze the following image and determine whether the image belongs to the smoking / cigarette category. Figure 5 ”.
[0088] Then, the thinking chain information corresponding to the sample data is used as the sample answer, including the category information of the sample data and the thinking chain information corresponding to the category information, so as to construct the following Figure 3 Question-answer pairs (QA pairs) are shown.
[0089] In a possible manner, the target data annotation model includes at least one of the following: a text-to-text model, a text-to-graph model, a graph-to-text model, a graph-to-text multimodal model, and a video-to-text multimodal model.
[0090] For example, the target data annotation model can be an AIGC model, such as a text-to-text model, a text-to-graph model, a graph-to-text model, a graph-to-text multimodal model, and a video-to-text multimodal model, etc., and the present disclosure does not limit this.
[0091] For example, the initial model can be trained based on the QA pair constructed above, so that the model learns to analyze the features in the image, make category judgments, and output the reasons for the judgment (thinking chain). Among them, the model training can include continue-pretrain, vision-text alignment, supervised fine-tuning, and reinforcement learning from human feedback. The required steps can be selected according to business needs, and the present disclosure does not limit this.
[0092] It is worth noting that the initial model already has a lot of common sense knowledge during the pre-training process, but does not understand professional knowledge. Therefore, if abstract professional knowledge and related labeled data are directly used, it is impossible to train a model that reaches or exceeds the level of manual labeling. Figure 7 , the professional knowledge in the labeling task can be decomposed into common sense, so that it is easier for the AIGC model to learn and understand, improve the model performance, and then construct corresponding sample data for the common sense rules, which can rely less on manually labeled data, and then train the initial model to obtain the target data labeling model.
[0093] In this way, in labeling systems that require the participation of professional labelers for a long time, the AIGC model can be used to perform labeling tasks, which can not only output category information but also the reasons for determining the category information, so that ordinary labelers can understand why the data belongs to this category, and then correct the labeling category to better optimize the model, so that the labeling accuracy of the AIGC model reaches or exceeds the accuracy of manual labeling, and the consistency rate of the AIGC model labeling is higher than the consistency rate of manual labeling, avoiding deviations, etc.
[0094] By adopting the above technical solution, the annotation tasks in professional fields can be transformed into annotation tasks based on common sense and can be quickly understood by ordinary annotators, and an efficient and high-performance AIGC model can be built to complete data annotation tasks in professional fields. In this way, annotation tasks can be completed based on common sense, and even for data in small languages, annotation tasks can be performed based on the model, improving the accuracy and efficiency of annotation.
[0095] Based on the same inventive concept, the present disclosure provides a data annotation method, referring to Figure 8, the method comprising:
[0096] S801. Obtain a question including target data.
[0097] S802: Input the question into the target data annotation model to obtain the answer to the question, which includes the category information of the target data and the target thinking chain information corresponding to the category information.
[0098] Among them, the target data annotation model is obtained by model training according to the training method of the above-mentioned data annotation model.
[0099] S803: Label the target data according to the category information.
[0100] By adopting the above method, data annotation is performed through the data annotation model, which is not affected by the professional level of the annotator, and there is no problem of inconsistent annotation categories for the same data. The accuracy and efficiency are higher than manual annotation. In addition, through the target thinking chain information corresponding to the category information, users can clearly understand the reasons for determining the category information.
[0101] Based on the same inventive concept, the present disclosure provides a training device for a data annotation model, referring to Fig. 9 , the device 900 comprises:
[0102] The first acquisition module 901 is used to acquire sample data that meets a preset labeling condition, wherein the preset labeling condition includes a single labeling condition obtained by decomposing a labeling rule corresponding to a labeling task, and the sample data corresponds to thought chain information, wherein the thought chain information is used to indicate whether the sample data belongs to a labeling category corresponding to the labeling task and the corresponding reason;
[0103] A first determination module 902 is used to determine a sample question corresponding to the sample data according to the marked category and the sample data, and to determine a sample answer corresponding to the sample data according to the thought chain information;
[0104] The training module 903 is used to perform model training based on the sample questions and the sample answers to obtain a target data labeling model. The target data labeling model is used to obtain answers to the questions based on the input questions including the target data. The answers include category information of the target data and target thinking chain information corresponding to the category information.
[0105] Optionally, the first acquisition module 901 is used to:
[0106] Obtain the sample data in at least one of the following ways:
[0107] Acquire first sample data that meets each of the single marking conditions and does not meet the exemption condition;
[0108] Acquire second sample data that simultaneously satisfies a plurality of the single marking conditions and does not satisfy the exemption condition;
[0109] Acquire third sample data that satisfies at least one of the single labeling conditions and the exemption condition, wherein the exemption condition represents a condition that the sample data is not labeled with the labeling category.
[0110] Optionally, the exemption conditions include at least one of the following conditions:
[0111] The sample data contains content indicating that the sample data does not belong to the marked category;
[0112] The sample data contains content of other categories except the marked category;
[0113] The data source of the sample data meets the preset conditions.
[0114] Optionally, the first acquisition module 901 is used to:
[0115] generating the sample data according to the preset annotation conditions and the data generation model; and / or,
[0116] The sample data is obtained by searching in a sample database according to the preset marking condition.
[0117] Optionally, the single marking condition is obtained in the following manner:
[0118] Obtaining the standard operating procedure document corresponding to the labeling task;
[0119] Decomposing the annotation rules for the annotation category in the standard operating procedure document to obtain a single annotation condition corresponding to the associated category of the annotation category; or,
[0120] The marking rules for the marking category in the standard operating procedure document are disassembled to obtain a single marking condition corresponding to a subcategory of the marking category, and a single marking condition corresponding to an associated category of the subcategory is determined.
[0121] Optionally, the thought chain information corresponding to the sample data is determined by at least one of the following methods:
[0122] Acquiring thought chain information manually configured for the sample data;
[0123] When the preset labeling conditions include multiple labeling conditions of different types, the target labeling conditions satisfied by the sample data are determined in the preset labeling conditions, and the thought chain information corresponding to the sample data is determined according to the target labeling conditions and the preset thought chain template;
[0124] The initial thinking chain information corresponding to the sample data is obtained by using a model for constructing thinking chain information, and the manual verification result of the initial thinking chain information is obtained. According to the manual verification result and the initial thinking chain information, the thinking chain information corresponding to the sample data is determined.
[0125] Optionally, the target data annotation model includes at least one of the following: a text-to-text model, a text-to-image model, a image-to-text model, a image-to-text multimodal model, and a video-to-text multimodal model.
[0126] Based on the same inventive concept, the present disclosure provides a data labeling device, referring to Fig.10 , the device 1000 comprises:
[0127] The second acquisition module 1001 is used to acquire questions including target data;
[0128] An input module 1002 is used to input the question into a target data annotation model to obtain an answer to the question, wherein the answer includes category information of the target data and target thinking chain information corresponding to the category information, wherein the target data annotation model is obtained by model training based on the training device of the above-mentioned data annotation model. An annotation module 1003 is used to annotate the target data according to the category information.
[0129] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0130] Based on the same concept, an embodiment of the present disclosure also provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processing device, the training method of the above-mentioned data annotation model or the steps of the data annotation method are implemented.
[0131] Based on the same concept, an embodiment of the present disclosure further provides an electronic device, including:
[0132] a storage device having a computer program stored thereon;
[0133] A processing device is used to execute the computer program in the storage device to implement the above-mentioned training method of the data annotation model or the steps of the data annotation method.
[0134] Reference below Fig.11, which shows a schematic diagram of the structure of an electronic device 1100 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.11 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0135] like Fig.11 As shown, the electronic device 1100 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage device 1108 to a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the electronic device 1100 are also stored. The processing device 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.
[0136] Typically, the following devices may be connected to the I / O interface 1105: an input device 1106 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1107 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1108 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1109. The communication device 1109 may allow the electronic device 1100 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.11 The electronic device 1100 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0137] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 1109, or installed from a storage device 1108, or installed from a ROM 1102. When the computer program is executed by the processing device 1101, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
[0138] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0139] In some embodiments, any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol) can be used for communication, and can be interconnected with any form or medium of digital data communication (e.g., communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), internets (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.
[0140] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0141] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains sample data that meets preset labeling conditions, wherein the preset labeling conditions include a single labeling condition obtained by decomposing the labeling rules corresponding to the labeling task, and the sample data corresponds to thought chain information, and the thought chain information is used to indicate whether the sample data belongs to the labeling category corresponding to the labeling task and the corresponding reason; determines the sample question corresponding to the sample data according to the labeling category and the sample data, and determines the sample answer corresponding to the sample data according to the thought chain information; performs model training according to the sample question and the sample answer to obtain a target data labeling model, and the target data labeling model is used to obtain the answer to the question according to the input question including the target data, and the answer includes the category information of the target data and the target thought chain information corresponding to the category information.
[0142] Alternatively, the computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a question including target data; inputs the question into a target data annotation model to obtain an answer to the question, wherein the answer includes category information of the target data and target thinking chain information corresponding to the category information, wherein the target data annotation model is obtained by model training according to the training method of the above-mentioned data annotation model; and annotates the target data according to the category information.
[0143] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0144] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0145] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a module does not, in some cases, limit the module itself.
[0146] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0147] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0148] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.
[0149] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0150] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
Claims
1. A method for training a data annotation model, characterized in that: The method comprises: Acquire sample data that meets a preset labeling condition, wherein the preset labeling condition includes a single labeling condition obtained by decomposing a labeling rule corresponding to a labeling task, and the sample data corresponds to thought chain information, wherein the thought chain information is used to indicate whether the sample data belongs to a labeling category corresponding to the labeling task and the corresponding reason; Determine a sample question corresponding to the sample data according to the marked category and the sample data, and determine a sample answer corresponding to the sample data according to the thought chain information; Model training is performed based on the sample questions and the sample answers to obtain a target data labeling model. The target data labeling model is used to obtain answers to input questions including target data, wherein the answers include category information of the target data and target thinking chain information corresponding to the category information.
2. The method according to claim 1, characterized in that The obtaining of sample data that meets the preset labeling conditions includes: Obtain the sample data in at least one of the following ways: Acquire first sample data that meets each of the single marking conditions and does not meet the exemption condition; Acquire second sample data that simultaneously satisfies a plurality of the single marking conditions and does not satisfy the exemption condition; Acquire third sample data that satisfies at least one of the single labeling conditions and the exemption condition, wherein the exemption condition represents a condition that the sample data is not labeled with the labeling category.
3. The method according to claim 2, characterized in that The exemption conditions include at least one of the following conditions: The sample data contains content indicating that the sample data does not belong to the marked category; The sample data contains content of other categories except the marked category; The data source of the sample data meets the preset conditions.
4. The method according to claim 1, characterized in that: The obtaining of sample data that meets the preset labeling conditions includes: generating the sample data according to the preset annotation conditions and the data generation model; and / or, The sample data is obtained by searching in a sample database according to the preset marking condition.
5. The method according to any one of claims 1 to 4, characterized in that: The single labeling condition is obtained in the following way: Obtaining the standard operating procedure document corresponding to the labeling task; Decomposing the annotation rules for the annotation category in the standard operating procedure document to obtain a single annotation condition corresponding to an associated category of the annotation category; or, The marking rules for the marking category in the standard operating procedure document are disassembled to obtain a single marking condition corresponding to a subcategory of the marking category, and a single marking condition corresponding to an associated category of the subcategory is determined.
6. The method according to any one of claims 1 to 4, characterized in that: The thought chain information corresponding to the sample data is determined by at least one of the following methods: Acquiring thought chain information manually configured for the sample data; When the preset labeling conditions include multiple labeling conditions of different types, the target labeling conditions satisfied by the sample data are determined in the preset labeling conditions, and the thought chain information corresponding to the sample data is determined according to the target labeling conditions and the preset thought chain template; The initial thinking chain information corresponding to the sample data is obtained by using a model for constructing thinking chain information, and the manual verification result of the initial thinking chain information is obtained. According to the manual verification result and the initial thinking chain information, the thinking chain information corresponding to the sample data is determined.
7. The method according to any one of claims 1 to 4, characterized in that: The target data annotation model includes at least one of the following: a text-to-text model, a text-to-graph model, a graph-to-text model, a graph-to-text multimodal model, and a video-to-text multimodal model.
8. A data labeling method, characterized in that: The method comprises: Get questions that include target data; Inputting the question into a target data annotation model to obtain an answer to the question, wherein the answer includes category information of the target data and target thought chain information corresponding to the category information, wherein the target data annotation model is obtained by model training according to the data annotation model training method according to any one of claims 1 to 7; The target data is labeled according to the category information.
9. A training device for a data annotation model, characterized in that: The device comprises: A first acquisition module is used to acquire sample data that meets a preset labeling condition, wherein the preset labeling condition includes a single labeling condition obtained by decomposing a labeling rule corresponding to a labeling task, and the sample data corresponds to thought chain information, wherein the thought chain information is used to indicate whether the sample data belongs to a labeling category corresponding to the labeling task and the corresponding reason; A first determination module is used to determine a sample question corresponding to the sample data according to the marked category and the sample data, and to determine a sample answer corresponding to the sample data according to the thought chain information; A training module is used to perform model training based on the sample questions and the sample answers to obtain a target data labeling model. The target data labeling model is used to obtain answers to input questions including target data, and the answers include category information of the target data and target thinking chain information corresponding to the category information.
10. A data labeling device, characterized in that: The device comprises: A second acquisition module, used for acquiring questions including target data; An input module, used for inputting the question into a target data annotation model to obtain an answer to the question, wherein the answer includes category information of the target data and target thought chain information corresponding to the category information, wherein the target data annotation model is obtained by model training according to the training device for the data annotation model according to claim 9; A labeling module is used to label the target data according to the category information.
11. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processing device, the steps of the method described in any one of claims 1 to 8 are implemented.
12. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 8.