A data labeling method, device, system, and storage medium

Abstract

The application provides a data labeling method, device and system and a storage medium, and belongs to the field of data labeling.The method comprises the following steps: S1, importing original training data and labeled data; S2, constructing an original labeling model, training the original labeling model according to the original training data and the labeled data to obtain a first labeling model; S3, importing unlabeled training data, predicting the unlabeled training data according to the first labeling model to obtain predicted data; and S4, analyzing the first labeling model according to the unlabeled training data and the predicted data to obtain a second labeling model.The application can realize the training of a pre-labeling model of a target field without the need for a large amount of manual labeling of text samples, greatly reduces the workload of manual labeling, saves the cost of data labeling work, and improves the accuracy of the model.

Description

Technical Field

[0001] This invention relates to the field of data annotation technology, specifically to a data annotation method, apparatus, system, and storage medium. Background Technology

[0002] In recent years, driven by both policy and technology, the artificial intelligence industry has experienced rapid growth, with data traffic continuing to increase rapidly. This has spurred a large demand for high-precision, scenario-based data, promoting the vigorous development of the data annotation industry. Text data annotation, in particular, is widely used in new retail, healthcare, customer service, advertising and marketing, social surveys and statistical analysis, and various aspects of daily life and entertainment.

[0003] As artificial intelligence (AI) further develops into specialized fields, algorithm research requires a large amount of labeled data from various domains for model training. This generates significant time and labor costs in data collection and manual annotation. Therefore, reducing the cost of data annotation is a crucial measure to promote the rapid implementation of AI across industries. One effective method to reduce data annotation costs is to pre-label the data, achieving semi-automation of data annotation and greatly reducing the workload of manual annotation.

[0004] In recent years, many studies have focused on Named Entity Recognition (NER) based on deep learning, and there has been considerable research in text pre-annotation. However, current research is often based on general scenarios, and the prediction models in these studies have the following shortcomings: because they cover a wide range of scenarios, the prediction models require a large number of unlabeled and labeled data samples for training, resulting in high human costs; when applied to specific domains, the accuracy of the models is not high enough, and the pre-annotation effect is not ideal; and the training effect of pre-annotated models is unsatisfactory due to the limited number of manually annotated data samples. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a data annotation method, apparatus, system and storage medium to address the shortcomings of the prior art.

[0006] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: A data annotation method, comprising the following steps:

[0007] S1: Import multiple raw training data and labeled data that correspond one-to-one with each of the raw training data;

[0008] S2: Construct an original annotation model by training the original annotation model based on multiple sets of original training data and the labeled data corresponding to the multiple sets of original training data to obtain a first annotation model;

[0009] S3: Import multiple unlabeled training data, predict each of the unlabeled training data according to the first labeling model, and obtain the predicted data of each of the unlabeled training data;

[0010] S4: Analyze the first labeled model based on the multiple unlabeled training data and the predicted data of the multiple unlabeled training data to obtain the second labeled model;

[0011] S5: Perform model analysis on the second labeled model based on the multiple original training data and the labeled data corresponding to the multiple original training data to obtain the third labeled model;

[0012] S6: Import the data to be tested, and annotate the data to be tested according to the third annotation model to obtain the data annotation results.

[0013] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: A data annotation device, comprising:

[0014] The data import module is used to import multiple raw training data and labeled data that correspond one-to-one with each of the raw training data.

[0015] The model training module is used to construct an original labeled model and train the original labeled model based on multiple original training data and the labeled data corresponding to the multiple original training data to obtain a first labeled model.

[0016] The prediction module is used to import multiple unlabeled training data, predict each of the unlabeled training data according to the first labeling model, and obtain the predicted data of each of the unlabeled training data.

[0017] The analysis module is used to analyze the first labeled model based on multiple unlabeled training data and the predicted data of the multiple unlabeled training data to obtain a second labeled model;

[0018] The model analysis module is used to perform model analysis on the second labeled model based on multiple sets of original training data and the labeled data corresponding to the multiple sets of original training data, so as to obtain a third labeled model;

[0019] The data annotation result acquisition module is used to import the data to be tested, annotate the data to be tested according to the third annotation model, and obtain the data annotation result.

[0020] Based on the above-mentioned data annotation method, the present invention also provides a data annotation system.

[0021] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: a data annotation system, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data annotation method described above is implemented.

[0022] Based on the above-described data annotation method, the present invention also provides a computer-readable storage medium.

[0023] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the data annotation method as described above.

[0024] The beneficial effects of this invention are as follows: a first annotation model is obtained by training the original annotation model with the original training data and the labeled data; predicted data is obtained by predicting the unlabeled training data based on the first annotation model; a second annotation model is obtained by training and analyzing the first annotation model with the unlabeled training data and the predicted data; a third annotation model is obtained by analyzing the second annotation model with the original training data and the labeled data; and data annotation results are obtained by annotating the test data with the third annotation model. This invention enables the training of a pre-annotated model in the target domain without requiring a large number of manually annotated text samples, greatly reducing the workload of manual annotation, saving the cost of data annotation, and improving the accuracy of the model. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating a data annotation method provided in an embodiment of the present invention;

[0026] Figure 2 A schematic diagram illustrating the structure of the pre-annotated model provided in an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram illustrating the training of a pre-labeled model provided in an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of machine pre-labeling based on a pre-labeled model provided in an embodiment of the present invention;

[0029] Figure 5 This is a schematic diagram illustrating the correction of the pre-annotated model provided in an embodiment of the present invention;

[0030] Figure 6 This is a block diagram of a data annotation device provided in an embodiment of the present invention. Detailed Implementation

[0031] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0032] Figure 1 This is a flowchart illustrating a data annotation method provided in an embodiment of the present invention.

[0033] like Figure 1 As shown, a data annotation method includes the following steps:

[0034] S1: Import multiple raw training data and labeled data that correspond one-to-one with each of the raw training data;

[0035] S2: Construct an original annotation model by training the original annotation model based on multiple sets of original training data and the labeled data corresponding to the multiple sets of original training data to obtain a first annotation model;

[0036] S3: Import multiple unlabeled training data, predict each of the unlabeled training data according to the first labeling model, and obtain the predicted data of each of the unlabeled training data;

[0037] S4: Analyze the first labeled model based on the multiple unlabeled training data and the predicted data of the multiple unlabeled training data to obtain the second labeled model;

[0038] S5: Perform model analysis on the second labeled model based on the multiple original training data and the labeled data corresponding to the multiple original training data to obtain the third labeled model;

[0039] S6: Import the data to be tested, and annotate the data to be tested according to the third annotation model to obtain the data annotation results.

[0040] It should be understood that a training sample set is input into the target domain, the training sample set containing a certain number of labeled sample texts (i.e., the labeled data) and a certain number of batches of unlabeled sample texts (i.e., the unlabeled training data).

[0041] Specifically, a batch of unlabeled sample texts (i.e., the unlabeled training data) from the training sample set is loaded, and the updated pre-labeled model (i.e., the first labeling model) is used to perform prediction processing to obtain the pre-labeled result (i.e., the prediction data).

[0042] In the above embodiments, a first annotation model is obtained by training the original annotation model with the original training data and the labeled data. Predicted data is obtained by predicting the unlabeled training data based on the first annotation model. A second annotation model is obtained by training and analyzing the first annotation model with the unlabeled training data and the predicted data. A third annotation model is obtained by analyzing the second annotation model with the original training data and the labeled data. Data annotation results are obtained by annotating the test data with the third annotation model. This method can achieve the training of a pre-annotated model in the target domain without the need for a large number of manually annotated text samples, which greatly reduces the workload of manual annotation, saves the cost of data annotation, and improves the accuracy of the model.

[0043] Optionally, as an embodiment of the present invention, the process of S2 includes:

[0044] A text convolutional neural network is constructed, and the text convolutional neural network is trained based on multiple sets of original training data and the labeled data corresponding to the multiple sets of original training data to obtain a first labeled model.

[0045] It should be understood that the TextCNN model, proposed by Yoon Kim in his paper "Convolutional Naural Networks for Sentence Classification," is a model that uses convolutional neural networks to handle NLP problems. Compared to traditional NLP models such as RNN / LSTM, CNNs can extract important features more efficiently, and these features play a crucial role in classification.

[0046] Specifically, based on the input sample text and the labeled entity classification information of the sample text, the labeled sample text (i.e., multiple original training data and the labeled data corresponding to the multiple original training data) is used to train the pre-labeled model (i.e., the text convolutional neural network), and the training results are updated to the dataset of the pre-labeled model to obtain a new pre-labeled model (i.e., the text convolutional neural network).

[0047] In the above embodiments, the first annotation model is obtained by training the text convolutional neural network based on the original training data and the labeled data. This can extract important features more efficiently, greatly reduce the workload of manual annotation, save the cost of data annotation, and improve the accuracy of the model.

[0048] Optionally, as an embodiment of the present invention, the process of S4 includes:

[0049] S41: Obtain the initial number of unlabeled training data, and count the total number of all unlabeled training data to obtain the total number of unlabeled training data;

[0050] S42: Sum the initial number of unlabeled training data with the total number of unlabeled training data to obtain the total number of unlabeled training data;

[0051] S43: Determine whether the total number of unlabeled training data is greater than or equal to a preset first total number. If not, execute S44; if yes, use the first labeled model as the second labeled model.

[0052] S44: Train the first labeled model based on multiple unlabeled training data and the predicted data of multiple unlabeled training data to obtain a fourth labeled model. Use the total number of unlabeled training data as the initial number of new unlabeled training data, and use the fourth labeled model as the new first labeled model, and return to S3.

[0053] Specifically, it is determined whether the batch size of unlabeled sample text used for training (i.e., the total number of unlabeled training data) reaches a preset value (i.e., the preset first total number).

[0054] If the result of the judgment is "no", that is, the preset batch number has not been reached, then the pre-labeled model (i.e. the first labeling model) is trained based on the entity classification information of the unlabeled sample text of the current batch and the pre-labeled results (i.e., multiple unlabeled training data and prediction data of multiple unlabeled training data), the training results are updated to the dataset of the pre-labeled model to obtain a new pre-labeled model (i.e. the fourth labeling model), and then a new batch of unlabeled sample text (i.e., multiple unlabeled training data are imported) is reloaded for training.

[0055] If the result of the judgment is "yes", then the first annotation model will be used as the second annotation model.

[0056] In the above embodiments, the second annotation model is obtained by analyzing the first annotation model based on multiple unlabeled training data and the prediction data of multiple unlabeled training data, which further improves the accuracy of the model. It can achieve the training of the pre-labeled model in the target domain without the need for a large number of manually labeled text samples, which greatly reduces the workload of manual annotation and saves the cost of data annotation.

[0057] Optionally, as an embodiment of the present invention, the process of S5 includes:

[0058] The second annotation model is used to annotate each of the original training data to obtain pre-annotated data for each of the original training data.

[0059] Verify whether the pre-labeled data of each original training data is the same as the labeled data corresponding to each original training data, and count the number of successful verifications to obtain the total number of successful verifications;

[0060] Determine whether the total number of successful verifications is greater than the preset second total number. If not, use the second annotation model as the new original annotation model and return to S1; if so, use the second annotation model as the third annotation model.

[0061] It should be understood that the updated pre-annotation model (i.e., the second annotation model) is used to pre-annotate the already annotated sample text (i.e., the original training data) in the training sample set, and the pre-annotation results (i.e., the pre-annotated data) are compared with the existing annotation results (i.e., the annotated data) of the samples.

[0062] Specifically, it is determined whether the comparison value between the pre-labeled result (i.e., the pre-labeled data) and the existing labeled result of the sample (i.e., the labeled data) meets the preset conditions.

[0063] If the judgment result is "no", then a new training sample set is input to conduct a new round of training on the pre-labeled model (i.e., the second labeled model). If the judgment result is "yes", then the program ends and the training of the pre-labeled model is completed.

[0064] In the above embodiments, a third annotation model is obtained by performing model analysis on the second annotation model based on multiple original training data and the labeled data corresponding to the multiple original training data. This enables the training of a pre-annotated model in the target domain without the need for a large number of manually annotated text samples, greatly reducing the workload of manual annotation, saving the cost of data annotation, and improving the accuracy of the model.

[0065] Optionally, as another embodiment of the present invention, the present invention includes:

[0066] S101, Input a training sample set for the target domain. The training sample set contains a certain number of labeled sample texts and a certain number of batches of unlabeled sample texts. S102, Load all labeled sample texts from the training sample set. S103, Train the pre-labeled model based on the input sample texts and their labeled entity classification information. Update the training results to the pre-labeled model's dataset to obtain a new pre-labeled model. S104, Load a batch of unlabeled sample texts from the training sample set. Use the updated pre-labeled model for prediction processing to obtain the pre-labeled results. S105, Determine if the batch size of unlabeled sample texts used for training has reached a preset value. If the result is "No," meaning the preset batch size has not been reached, proceed to S106, based on the current batch... The system uses unlabeled sample text and pre-labeled entity classification information to train a pre-labeled model. The training results are then updated to the pre-labeled model's dataset, resulting in a new pre-labeled model. The process then returns to S104 to load a new batch of unlabeled sample text for training. If the result of S105 is "yes," then S107 is executed, using the updated pre-labeled model to pre-label the labeled sample text in the training sample set, comparing the pre-labeled results with the existing labeling results. In S108, it checks whether the comparison between the pre-labeled results and the existing labeling results meets a preset condition. If the result is "no," the process returns to S101, inputting a new training sample set for a new round of training. If the result is "yes," the program ends, completing the pre-labeled model training. This method enables the training of a pre-labeled model for the target domain without requiring a large number of manually labeled text samples. The trained pre-labeled model can be applied to text entity classification data annotation in the target domain. Through machine annotation and manual correction, the workload of manual annotation is greatly reduced, saving costs associated with data annotation.

[0067] Optionally, as another embodiment of the present invention, the unlabeled text of the present invention may be obtained from knowledge graph collection and processing based on the target domain, while the labeled sample text is obtained by manually labeling some unlabeled text samples.

[0068] Alternatively, as another embodiment of the present invention, such as Figure 2As shown, the pre-annotation model of this invention consists of a model training algorithm, a model correction algorithm, a pre-annotation algorithm, and a dataset. The dataset includes three parts: a mapping relationship between general entity classification and domain entity classification, entity and domain entity classification, and feature words and frequencies. The mapping relationship between general entity classification and domain entity classification supports the conversion of entity classifications obtained from entity recognition into domain model classifications during pre-annotation. The entity and domain entity classification records the entity and entity classification key-value pairs within the target domain obtained from model training. The feature words and frequencies record the feature words contained in the text content and their frequencies during training through a feature word table. Entity recognition relies on the feature word table.

[0069] Alternatively, as another embodiment of the present invention, such as Figure 3 As shown, this invention uses a pre-labeled model training algorithm for model training. This algorithm extracts entities and general entity classifications from manually labeled text data or machine-labeled pre-labeled text data. Based on comparison with existing labeling results, it maps the extracted general entity classifications to the classifications set for the target domain in the current pre-labeled model, updating the pre-labeled model's dataset. If the same record already exists in the dataset, its count is incremented. Simultaneously, the obtained entity-domain entity classification relationships are updated in the pre-labeled model's dataset. For identical or similar entity classifications that appear repeatedly in different samples, the count of existing entity-domain entity classification relationship records is incremented. Furthermore, the already labeled entities are used as feature words, updating the feature words and their frequencies in the pre-labeled model's dataset.

[0070] Alternatively, as another embodiment of the present invention, such as Figure 4 As shown, this invention utilizes a pre-labeling algorithm within an updated pre-labeling model. Unlabeled text data is input into the pre-labeling model, and combined with the dataset of the pre-labeling model, the pre-labeling algorithm is used to predict the unlabeled text data to obtain pre-labeling results. The implementation sub-steps of the pre-labeling algorithm include: S104-01 Extracting entities and general entity classifications from the unlabeled text data, and supplementing the entity extraction with feature words and frequencies from the pre-labeling model; S104-02 If the extracted entity already exists in the entity and domain entity classification records in the dataset, the existing entity and domain entity classification records are used to obtain the pre-labeling result, and the pre-labeling model is updated for this entity, increasing the count of entity and domain entity classifications in the dataset; S104-03 If the extracted entity does not exist in the entity and domain entity classification records in the dataset, the possible domain entity classification is predicted based on the mapping relationship and frequency between general entity classification and domain entity classification to obtain the pre-labeling result.

[0071] Alternatively, as another embodiment of the present invention, such as Figure 5As shown, the present invention validates the pre-labeled model by pre-labeling a certain number of manually labeled sample texts in the training sample set. The correctness of the pre-labeling results is judged based on the manually labeled results, and the pre-labeling accuracy of all extracted entities is calculated to determine whether a preset condition is met. If the pre-labeling accuracy meets the preset condition, the program ends, and the training of the pre-labeled model is completed. If the accuracy does not meet the preset condition, the pre-labeled model is retrained using a new batch of datasets.

[0072] Preferably, the correction of the pre-labeled model utilizes a model correction algorithm within the pre-labeled model. The deviation between the machine pre-labeling and manual correction results is obtained. The target text and the deviation between the manual and pre-labeling results of the target text are then input into the pre-labeled model. The model correction algorithm then corrects the model's dataset based on this deviation.

[0073] Optionally, as another embodiment of the present invention, the beneficial effects of the present invention are as follows: Annotated text in the target domain and a certain number of batches of unannotated text are input into a pre-annotation model. The model annotates the entity types of the unannotated text in the target domain according to the entity types of the annotated text, and updates the annotated entity types of the unannotated text in the target domain to the dataset of the pre-annotation model, thereby updating the model. Then, the pre-annotation algorithm of the pre-annotation model predicts the entity types of a batch of unannotated text, obtaining the predicted entity types. The pre-annotated text is then added to the annotated text dataset for the pre-annotation of the next batch of unannotated text. This step is repeated until training of all batches of unannotated text is completed. Finally, the updated pre-annotation model is used to pre-annotate the annotated sample text in the sample set. The pre-annotation results are compared with the annotation results provided by the samples. Whether retraining is performed is determined based on whether the comparison value reaches a preset value, and an entity annotation model is obtained. When using the pre-annotation model to assist manual annotation, the pre-annotation model can be corrected by manually modifying the pre-annotation results. This system enables reliable pre-labeled models to be trained using both labeled and unlabeled text from the target domain without human intervention. Applying these pre-labeled models to data pre-labeling transforms the manual annotation process into a model of machine pre-labeling followed by manual correction, reducing the workload of manual text entity annotation. Simultaneously, manually corrected data is used to refine and optimize the pre-labeled models. Different domain-specific text datasets can be used to train respective pre-labeled models, reducing the amount of text data required for training while enhancing the accuracy of the pre-labeled models in the target domain.

[0074] Figure 6 This is a block diagram of a data annotation device provided in an embodiment of the present invention.

[0075] Alternatively, as another embodiment of the present invention, such as Figure 6 As shown, a data annotation device includes:

[0076] The data import module is used to import multiple raw training data and labeled data that correspond one-to-one with each of the raw training data.

[0077] The model training module is used to construct an original labeled model and train the original labeled model based on multiple original training data and the labeled data corresponding to the multiple original training data to obtain a first labeled model.

[0078] The prediction module is used to import multiple unlabeled training data, predict each of the unlabeled training data according to the first labeling model, and obtain the predicted data of each of the unlabeled training data.

[0079] The analysis module is used to analyze the first labeled model based on multiple unlabeled training data and the predicted data of the multiple unlabeled training data to obtain a second labeled model;

[0080] The model analysis module is used to perform model analysis on the second labeled model based on multiple sets of original training data and the labeled data corresponding to the multiple sets of original training data, so as to obtain a third labeled model;

[0081] The data annotation result acquisition module is used to import the data to be tested, annotate the data to be tested according to the third annotation model, and obtain the data annotation result.

[0082] Optionally, as an embodiment of the present invention, the model training module is specifically used for:

[0083] A text convolutional neural network is constructed, and the text convolutional neural network is trained based on multiple sets of original training data and the labeled data corresponding to the multiple sets of original training data to obtain a first labeled model.

[0084] Optionally, as an embodiment of the present invention, the analysis module is specifically used for:

[0085] Obtain the initial number of unlabeled training data, and count the total number of all unlabeled training data to obtain the total number of unlabeled training data.

[0086] The initial number of unlabeled training data is summed with the total number of unlabeled training data to obtain the total number of unlabeled training data.

[0087] Determine whether the total number of unlabeled training data is greater than or equal to a preset first total number. If not, train the first labeling model based on multiple unlabeled training data and the prediction data of multiple unlabeled training data to obtain a fourth labeling model. Use the total number of unlabeled training data as the new initial number of unlabeled training data, and use the fourth labeling model as the new first labeling model, and return to the prediction module. If yes, use the first labeling model as the second labeling model.

[0088] Optionally, as an embodiment of the present invention, the model analysis module is specifically used for:

[0089] The second annotation model is used to annotate each of the original training data to obtain pre-annotated data for each of the original training data.

[0090] Verify whether the pre-labeled data of each original training data is the same as the labeled data corresponding to each original training data, and count the number of successful verifications to obtain the total number of successful verifications;

[0091] Determine whether the total number of successful verifications is greater than the preset second total number. If not, use the second annotation model as the new original annotation model and return to the data import module; if so, use the second annotation model as the third annotation model.

[0092] Optionally, another embodiment of the present invention provides a data annotation system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the data annotation method described above. This system can be a computer or similar system.

[0093] Optionally, another embodiment of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the data annotation method described above.

[0094] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0095] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0096] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0097] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0098] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0099] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data annotation method, characterized in that, Includes the following steps: S1: Import multiple raw training data and labeled data that correspond one-to-one with each of the raw training data, wherein the labeled data is labeled sample text; S2: Construct an original annotation model by training the original annotation model based on multiple sets of original training data and the labeled data corresponding to the multiple sets of original training data to obtain a first annotation model; S3: Import multiple unlabeled training data, predict each unlabeled training data according to the first labeling model, and obtain the predicted data of each unlabeled training data, wherein the unlabeled training data is unlabeled sample text; S4: Analyze the first labeled model based on the multiple unlabeled training data and the predicted data of the multiple unlabeled training data to obtain the second labeled model; S5: Perform model analysis on the second labeled model based on the multiple original training data and the labeled data corresponding to the multiple original training data to obtain the third labeled model; S6: Import the data to be tested, and annotate the data to be tested according to the third annotation model to obtain the data annotation results; The process in S4 includes: S41: Obtain the initial number of unlabeled training data, and count the total number of all unlabeled training data to obtain the total number of unlabeled training data; S42: Sum the initial number of unlabeled training data with the total number of unlabeled training data to obtain the total number of unlabeled training data; S43: Determine whether the total number of unlabeled training data is greater than or equal to a preset first total number. If not, execute S44; if yes, use the first labeled model as the second labeled model. S44: Train the first labeled model based on multiple unlabeled training data and the predicted data of multiple unlabeled training data to obtain a fourth labeled model. Use the total number of unlabeled training data as the initial number of new unlabeled training data, and use the fourth labeled model as the new first labeled model, and return to S3.

2. The data annotation method according to claim 1, characterized in that, The process of S2 includes: A text convolutional neural network is constructed, and the text convolutional neural network is trained based on multiple sets of original training data and the labeled data corresponding to the multiple sets of original training data to obtain a first labeled model.

3. The data annotation method according to claim 1, characterized in that, The process of S5 includes: The second annotation model is used to annotate each of the original training data to obtain pre-annotated data for each of the original training data. Verify whether the pre-labeled data of each original training data is the same as the labeled data corresponding to each original training data, and count the number of successful verifications to obtain the total number of successful verifications; Determine whether the total number of successful verifications is greater than the preset second total number. If not, use the second annotation model as the new original annotation model and return to S1; if so, use the second annotation model as the third annotation model.

4. A data annotation device, characterized in that, include: The data import module is used to import multiple raw training data and labeled data that correspond one-to-one with each of the raw training data, wherein the labeled data is labeled sample text. The model training module is used to construct an original labeled model and train the original labeled model based on multiple original training data and the labeled data corresponding to the multiple original training data to obtain a first labeled model. The prediction module is used to import multiple unlabeled training data, predict each of the unlabeled training data according to the first labeling model, and obtain the predicted data of each of the unlabeled training data, wherein the unlabeled training data is unlabeled sample text; The analysis module is used to analyze the first labeled model based on multiple unlabeled training data and the predicted data of the multiple unlabeled training data to obtain a second labeled model; The model analysis module is used to perform model analysis on the second labeled model based on multiple sets of original training data and the labeled data corresponding to the multiple sets of original training data, so as to obtain a third labeled model; The data annotation result acquisition module is used to import the data to be tested, annotate the data to be tested according to the third annotation model, and obtain the data annotation result; The model analysis module is specifically used for: The second annotation model is used to annotate each of the original training data to obtain pre-annotated data for each of the original training data. Verify whether the pre-labeled data of each original training data is the same as the labeled data corresponding to each original training data, and count the number of successful verifications to obtain the total number of successful verifications; Determine whether the total number of successful verifications is greater than the preset second total number. If not, use the second labeled model as the new original labeled model and return to the data import module. If so, the second annotation model will be used as the third annotation model.

5. The data annotation device according to claim 4, characterized in that, The model training module is specifically used for: A text convolutional neural network is constructed, and the text convolutional neural network is trained based on multiple sets of original training data and the labeled data corresponding to the multiple sets of original training data to obtain a first labeled model.

6. The data annotation device according to claim 4, characterized in that, The analysis module is specifically used for: Obtain the initial number of unlabeled training data, and count the total number of all unlabeled training data to obtain the total number of unlabeled training data. The initial number of unlabeled training data is summed with the total number of unlabeled training data to obtain the total number of unlabeled training data. If the total number of unlabeled training data is greater than or equal to a preset first total number, then the first labeling model is trained based on the multiple unlabeled training data and the prediction data of the multiple unlabeled training data to obtain a fourth labeling model. The total number of unlabeled training data is used as the new initial number of unlabeled training data, and the fourth labeling model is used as the new first labeling model. The result is returned to the prediction module. If the result is yes, then the first labeling model is used as the second labeling model.

7. A data annotation system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the data annotation method as described in any one of claims 1 to 3.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the data annotation method as described in any one of claims 1 to 3.

Images