Label marking method, device and medium

Abstract

The present invention relates to the field of data processing technology, and is particularly suitable for artificial intelligence data labeling. In view of the situation that there are uncertain labels, multiple labels, and labels increase with application scenarios when labeling initial data in actual data processing, the label labeling method, device and medium of the present invention are proposed, aiming to solve the technical problem of how to achieve multi-label labeling for data with uncertain labels. To this end, the method of the present invention manually labels multiple uncertain labels through random sampling data, and after modeling training and sampling inspection through label prediction probability, the unlabeled data within the range is added to the expanded data according to the accuracy, and thus enters the next round of repeated modeling training to expansion process until all data have completed the labeling of one or more dynamically added and changed labels. Thereby, automatic multi-label labeling of massive data using machine learning is realized, which improves efficiency, reduces costs, and reduces manual burden, and helps to improve user experience.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and is particularly suitable for intelligent data automatic analysis and labeling, and more specifically relates to a labeling method, device and medium. Background Art

[0002] With the development of technologies such as big data, cloud computing, networks (not limited to the Internet / communication networks), and machine learning, various information data are growing at an exponential rate, especially in multiple industries through accurate and effective analysis and processing of massive data in network-based interactions, production and life, and artificial intelligence. The optimization of the analysis and processing of big data can often quickly improve the efficiency and advantages of the industry. For example, in the recognition and processing of text, voice, and / or images, the automatic analysis and annotation of massive data can improve the training efficiency of machine learning algorithms that need to rely on massive data samples for training, make more accurate predictions, reduce errors, and reduce the complexity of data processing. Especially in supervised learning, by building machine learning algorithm models and training algorithms, the data types of various industries are complex and the data dimensions are high, which poses huge challenges for tasks such as data classification and labeling. The labeling cost is high, and new classifications and new labels are often generated for data with changes in business, scenarios, platform requirements, etc. The dynamic change of labels further increases the difficulty and cost of labeling. For example: for the same paragraph of text, the label cannot be determined at the beginning, that is, the label is uncertain, it can be A, B, C, etc., and it may also be A, B, C, etc., and it may also be a multi-class label.

[0003] Therefore, it is necessary to provide a solution that can perform multi-label annotation for data with uncertain labels and intelligently perform automatic analysis and annotation, improve the scalability of label and data annotation, and multi-label annotation for large amounts of data can effectively reduce the amount of manual annotation, reduce costs and labor burden, dynamically add labels and corresponding annotations, and thus improve user experience. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] The present invention aims to solve the technical problem of how to achieve multi-label annotation for data with uncertain labels; further, to solve the technical problem of how to use the manually annotated labels of a small amount of data to complete the annotation of all the remaining huge data after machine learning; further, to solve the technical problem of how to update the extended labels during annotation and annotate the data.

[0006] (II) Technical solution

[0007] To solve the above technical problems, the first aspect of the present invention proposes a labeling method, comprising: training a prediction model through labeled data in a training set, and predicting unlabeled data through the prediction model; extracting sample data for inspection based on the prediction results, and deciding whether to add new labels; checking the stratification of labels predicted by the prediction model to determine whether it is necessary to label the unlabeled data and add it to the training set; repeating the training steps to the step of adding to the training set until the expected labeling conditions are met.

[0008] According to a preferred embodiment of the present invention, the prediction model is specifically a model constructed by xBERT; the loss function during the training of the prediction model includes a cross-entropy loss function calculated from two parts: data manually annotated by the user and data annotated by model extension; the labeled data is obtained by randomly sampling an initial fixed number of data, and the user manually annotates the sampled data with one or more labels; and the data initially annotated with one or more labels is added to the training set.

[0009] According to a preferred embodiment of the present invention, it also includes: before training the prediction model, first determine whether part of the data input to the prediction model has been labeled; if so, directly input it into the prediction model for model training; if not, it means that no data has been labeled, and it is necessary to randomly sample an initial fixed number of data from the data input to the prediction model for the initial labeling.

[0010] According to a preferred embodiment of the present invention, sample data is extracted for inspection based on the prediction results, which specifically includes: based on one or more label prediction probabilities output by the prediction model, sample data is extracted from the category corresponding to each label for inspection; the extracted sample data specifically includes at least one or more of the following sampling: stratified sampling according to label prediction probability, vocabulary diversity sampling, random supplementary sampling, machine-labeled data model score jitter sampling; the inspection specifically includes: the user manually verifies the sample data to label the sample data, and determines whether to add one or more new labels based on the verification.

[0011] According to a preferred embodiment of the present invention, it also includes: for long-tail unbalanced labels, a special label "others" is added, and the special label is excluded in the vocabulary diversity sampling and random sampling process to increase the probability of extracting potential positive examples; and / or, the vocabulary diversity sampling refers to the vocabulary division in all data, which is extracted from unlabeled data that is significantly different from the vocabulary division of labeled data; and / or, the machine-labeled data model score jitter sampling is used, and the standard deviation of the historical prediction score of the extended labeled label for the unlabeled data is counted. If the standard deviation exceeds a specific threshold, it is used as the extracted sample data to deal with potential erroneous labels in the extended labeling.

[0012] According to a preferred embodiment of the present invention, the stratification of labels predicted by the prediction model is checked to determine whether it is necessary to label the data without labels and add them to the training set, specifically including: checking the accuracy of the predicted label stratification, if the stratification level reaches a preset accuracy threshold, all the data without labels within the range of the level are labeled and added to the training set to expand the labeled data.

[0013] According to a preferred embodiment of the present invention, the training steps up to the step of adding to the training set are repeated until the expected labeling conditions are met, specifically including: before repeating the training steps, statistics are performed on the labeled data to determine whether the expected coverage is achieved; if so, the expansion of the labeled data is ended; if not, repeated cycle training of the prediction model is started; and, when there are one or more new labels added, the prediction model adds predictions for the new one or more labels in training and prediction.

[0014] According to a preferred embodiment of the present invention, it also includes: when training the prediction model, if the labels in the manually labeled data are relatively balanced, the manually labeled data and the model expanded labeled data are used; if the labels in the manually labeled data are unbalanced, the manually inspected data are selected as the model training data.

[0015] To solve the above technical problems, the second aspect of the present invention proposes a label marking device, including: a training prediction module, used to train a prediction model through labeled data in a training set, and to predict unlabeled data through the prediction model; a sampling inspection module, used to extract sample data for inspection according to the prediction results, and decide whether to add new labels; an extended annotation module, used to check the stratification of labels predicted by the prediction model to determine whether it is necessary to label the unlabeled data and add it to the training set; a return repetition module, used to repeat the training steps to the step of adding to the training set until the expected labeling conditions are met.

[0016] To solve the above technical problems, the third aspect of the present invention proposes an electronic device, comprising a processor and a memory, wherein the memory is used to store a computer executable program, and when the computer program is executed by the processor, the processor executes the method described in any one of the first aspects.

[0017] In order to solve the above technical problems, the fourth aspect of the present invention proposes a computer-readable medium storing a computer executable program, and when the computer executable program is executed, it implements the method described in any one of the first aspects.

[0018] In order to solve the above technical problem, the fifth aspect of the present invention proposes a computer executable program, which, when executed, implements the method described in any one of the first aspects.

[0019] (III) Beneficial effects

[0020] The present invention aims at the situations in which the data labeling workload in massive data processing is large, the labels cannot be determined in the initial labeling (i.e., the labels are uncertain), the data (text, voice, image and other data) may actually have multiple labels belonging to multiple categories, and the labels increase with the application scenarios. By randomly sampling a small amount of data, multiple uncertain labels are manually labeled, and after machine learning modeling training and prediction and label expansion, the modeling and label expansion process is repeated until all data labels are labeled, thereby realizing automatic labeling by autonomous learning.

[0021] Furthermore, by manually completing multi-label annotation of a small amount of data in the initial state of uncertain labels, modeling training and prediction based on this, stratified sampling and inspection according to the label prediction probability, expanding the multi-label annotation of unlabeled data or adding labels according to the inspection results, and thus entering the next round of repeated modeling training to expansion process until all data have completed the annotation of one or more dynamically added and changed labels. In this way, automatic multi-label annotation and inspection are realized for data with uncertain labels, and the annotation of all the remaining data is completed after machine learning using only the manually annotated labels of a small amount of data, realizing efficient automatic multi-label annotation of massive data, reducing costs and alleviating the labor burden.

[0022] Furthermore, when annotating, the extended tags are updated and the data is annotated. This method of dynamically adding tags and multi-tag annotation is in line with the development trend of data processing, adapts to more application scenarios and industry businesses, has strong data processing scalability, and helps to improve user experience.

[0023] Furthermore, the combination of probability stratified sampling strategy and label prediction probability can further improve the efficiency of intelligent labeling of complex and large data. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1is a main flow chart of an embodiment of a label marking method according to the present invention;

[0025] Figure 2 is a main structural block diagram of an embodiment of a label marking device according to the present invention;

[0026] Figure 3 is a main structural block diagram of an embodiment of an electronic device according to the present invention;

[0027] Figure 4 is a schematic diagram of the main structure of an embodiment of a more specific electronic device according to the present invention;

[0028] Figure 5 is a main structural diagram of an embodiment of a computer-readable medium according to the present invention;

[0029] Figure 6 is a specific flow chart of an embodiment of the technical solution according to the present invention;

[0030] Figure 7 It is a schematic diagram of a front-end user manual annotation interface according to an embodiment of the technical solution of the present invention. DETAILED DESCRIPTION

[0031] In the process of introducing specific embodiments, the detailed description of the structure, performance, effect or other characteristics is to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art can implement the present invention with a technical solution that does not contain the above-mentioned structure, performance, effect or other characteristics under certain circumstances.

[0032] The flowchart in the accompanying drawings is only an exemplary process demonstration, and does not mean that the solution of the present invention must include all the contents, operations and steps in the flowchart, nor does it mean that it must be executed in the order shown in the figure. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. The execution order shown in the flowchart can be changed according to actual conditions without departing from the main purpose of the present invention.

[0033] Frames in the attached figure Figure 1 Generally, it refers to functional entities, which do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processing unit devices and / or microcontroller devices.

[0034] The same reference numerals in the drawings represent the same or similar elements, components or parts, and thus the repeated description of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other attributives representing the numbers may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these attributives. In other words, these attributives are only used to distinguish one from another. For example, the first device may also be called the second device, but it does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0035] The following first describes some technical terms that may be involved in the embodiments of the present invention:

[0036] Data labeling: refers to the process of processing artificial intelligence learning data by data processing personnel using labeling tools such as BasicFinder. Common types of data labeling include image labeling, voice labeling, text labeling, video labeling, etc. The basic forms of labeling include labeling frames, 3D frames, text transcription, image dotting, target object contours, etc.

[0037] The main implementation process of an embodiment of the present invention is, for example: S1, a prediction model is constructed by training a small amount of labeled data and predicting unlabeled data; S2, sample data is extracted for inspection based on the prediction results, and it is determined whether to add new labels; S3, unlabeled data is labeled based on the inspection results; S1 to S3 are repeated until all unlabeled data are labeled.

[0038] [Example 1]

[0039] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the method implementation of the present invention is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0040] Here we will combine Figure 1 The main steps of an embodiment of the method of the present invention are described in the flowchart shown in FIG. The method mainly includes:

[0041] Step S110, training a prediction model using labeled data in a training set, and predicting unlabeled data using the prediction model;

[0042] Step S120, extracting sample data for inspection based on the prediction results, and deciding whether to add a new label;

[0043] Step S130, checking the accuracy of the predicted label stratification to determine whether all unlabeled data at the stratification level that reaches a preset accuracy threshold are labeled and added to the training set, thereby continuing the model expansion (i.e., machine expansion annotation);

[0044] Step S140, repeat the training steps to the step of adding to the training set until all unlabeled data are labeled.

[0045] In one embodiment of step S110, the prediction model is trained by using labeled data in the training set, specifically including: the labeled data can be obtained by: randomly sampling one or more data to be processed, providing them to the front end, and having the user manually perform initial labeling of one or more labels on the sampled data.

[0046] Furthermore, when the user performs manual annotation, one or more existing tags recommended to the user may be displayed on the interactive interface of the application performing the annotation.

[0047] Further, the data initially labeled with one or more labels is added to the training set. And / or, the prediction model is specifically a model built by xBERT. Taking the multi-label labeling method with uncertain labels in the text data processing process as an example:

[0048] Preprocessing to obtain text data: preliminarily screen the text information and remove invalid text to obtain the preliminarily screened text; further perform word segmentation, stop word removal, statistics, etc. on the preliminarily screened text to obtain valid text data (valid word list). Even, cluster the texts, perform statistics to obtain the similarity between texts and representative texts, extract a certain amount of representative texts from the text collection according to clustering clusters, valid words, etc., and annotate them.

[0049] Perform initial annotation: randomly select one or more data to be processed from the text data. If the data to be processed has not been labeled before, it will be manually labeled by the user, which is the initial annotation. If some of the data have been labeled, it can directly become the training data of the model training set. Among them, the random sampling can pre-set a fixed number of sampled data, such as 200, 300, etc.

[0050] For example, in the case of manual annotation by users, the sentence (data to be processed) is "EU rejects German call to boycottBritish lamb.". At this time, the user enters the pre-defined three labels A, B, and C. When the user manually annotates the sentence, it is judged to belong to the three categories of A and B, and it is found that there should be a new label D, so the D label is continued to be entered, and the three labels A, B, and D are annotated. For other unlabeled data, it is not certain which label or labels should be annotated, so the labels are uncertain and intelligent automatic multi-label annotation is realized. The annotation labels can use various existing or self-developed rapid annotation platforms. The pre-defined labels such as A, B, and C can also be extracted from the historical labels of related projects. Each piece of text data uploaded by sampling is annotated, and the labeled data is exported.

[0051] Example 1:

[0052] An input file might look like this: import.json

[0053] {"text":"EU rejects German call to boycott British lamb.","meta":{"external_id":1}}

[0054] The exported file is as follows: output.json

[0055] {"doc_id":2023,"text":"EU rejects German call to boycott Britishlamb.","labels":["news"],["Trade"],["EU"]"}

[0056] Example 2:

[0057] Text category annotation:

[0058] Id: 1; What is the form of counter guarantee; label: 4, 7

[0059] Id: 2; What is the bank loan procedure; label: 4, 9

[0060] …

[0061] Example 3:

[0062] Keyword annotation of text data:

[0063] Id: 127; Compensation for vehicle damage on residential roads

[0064] Id: 129; Compensation for vehicle accident property damage liability disputes

[0065] …

[0066] Obtaining a training set: The labeled text data, including the text data initially labeled with one or more labels by the aforementioned user, is formed into a training set or added to the training set.

[0067] Model learning: training the established prediction model. The prediction model can be the algorithm part of the active learning platform that can perform automatic labeling. Specifically, xBERT can be used to build a model, two-way learning (NLP), and more accurate prediction. The training and modeling of the xBERT model for labeled data will not be described here. Preferably, the strategy that can be adopted for the labeled data used in model modeling: (1) If the labels are relatively balanced in the manually labeled data, then use manual inspection + machine expansion data; (2) If the labels are relatively unbalanced in the manually labeled data, then select the manually inspected data as the model training data. Preferably, the loss function for xBERT model training can adopt the cross entropy function loss, which consists of two parts: the first part is the manually labeled part, and the second part w mechine The part of the labeled data expanded for the machine, the two together serve as the loss function of the overall model. For example:

[0068]

[0069] Among them, the machine-expanded data is regarded as a positive example of the corresponding label as a whole, and its contribution to the loss function is adjustable; c It is a hyperparameter for adjusting the recall rate Recall and precision Precision. If it is greater than 1, it is biased towards Recall; n, z represent the total number of samples, c, represents the sample, y n,c represents the actual label, σ(x n，c ), represents the predicted output. Furthermore, the second part only takes data, can cope with the situation where label uncertainty increases.

[0070] In one embodiment, the prediction model is used to predict the unlabeled data, specifically including: inputting the unlabeled data and the data with uncertain labels into the xBERT model, and predicting the prediction results corresponding to the output data, such as the label prediction probability, etc. For example, the label A prediction probability of text data 1 is 0.6, the label B prediction probability is 0.3, and the label C prediction probability is 0.1.

[0071] It can be seen that by combining manual labeling with machine learning model training and prediction, as much data as possible is predicted and labeled, and the labeled data is accumulated and added to the training set to train and update the model, multi-label labeling with uncertain labels is achieved. With a small amount of manual multi-label labeling, multi-label labeling of a large amount of data can be completed by autonomous learning, which improves labeling efficiency, reduces the burden of manual labeling, and reduces the cost of tedious work.

[0072] In one embodiment of step S120, extracting sample data for inspection according to the prediction result specifically includes: extracting sample data from the category corresponding to each label for inspection according to one or more label prediction probabilities output by the prediction model, and deciding whether to add a new label.

[0073] Further, the extracting of sample data specifically includes: extracting one or more sample data from the category corresponding to each label according to the label prediction probability and the predetermined sampling quantity. For example, stratified sampling is performed from all data according to the label prediction probability. Generally, there can be multiple strategies for extracting samples. In this strategy set, the optimal strategy is determined and recommended based on the feature analysis of the data currently labeled. For example, in the case of predicting labels for input data, it is mainly to determine the probability of whether to label a certain label. Preferably, the probability can be recommended to do stratified sampling strategy. The sampling process can refer to the existing sampling principles and will not be repeated here. Preferably, it includes but is not limited to one or more of the following sampling strategies: stratified sampling, vocabulary diversity sampling, random supplementary sampling, and machine-labeled data model score jitter sampling can be performed according to the label prediction probability. Further, the sampling strategy can be in the above order or can be changed in any combination. Among them, the vocabulary diversity sampling is adopted to consider the vocabulary division in all data and extract from the unlabeled data with a large difference from the vocabulary division of the labeled data. In addition, in order to deal with potential erroneous labels in the machine label expansion process, the machine label data model score jitter sampling is used, and the standard deviation of the historical prediction scores of the expanded labels of the data is calculated. If the standard deviation exceeds a specific threshold, the sampled data can be added for manual review.

[0074] Furthermore, in sampling, we can add a special tag "others" to target long-tail imbalanced tags, and exclude this special tag in vocabulary diversity sampling and random sampling processes to increase the probability of extracting potential positive examples.

[0075] In one embodiment, the inspection specifically includes: the user manually verifies the sample data. Furthermore, the extracted sample data includes or does not include data without labels, and the manual analysis verifies whether one or more labels corresponding to the data are accurate and correct, and then whether the labeling meets the requirements or satisfies various preset conditions. For example, the inspection process is a labeling process for the front-end user to determine whether the data is labeled, whether the labeling is incorrect, whether the labeling is complete / which labels are labeled, etc. based on the sampled data and labels provided on the interactive interface.

[0076] In one implementation, the user can manually check and inspect the extracted sample data to determine whether a new label needs to be added, and the new label added can be one or more.

[0077] It can be seen that automatic multi-label annotation is implemented for data with uncertain labels and its correctness is verified, which helps to improve the accuracy of subsequent multi-label annotation of data, thereby helping to update the constructed model to provide better predictions and autonomous annotations. The expansion of autonomous annotation based on machine learning can adapt to data annotation in more and wider application fields. In addition, sampling according to the label prediction probability and the coordination of label prediction probability can further improve the efficiency of intelligent annotation of complex and large data. By checking the sampled data, users can determine the accuracy and correctness of the model and decide whether to add one or more new labels.

[0078] In one embodiment of step S130, the accuracy of the predicted label stratification is checked to determine whether all unlabeled data that have reached a preset accuracy threshold are labeled and added to the training set. Specifically, it includes: checking the accuracy of the label stratification predicted by the model in the background. If the level reaches the preset accuracy value of 0.9, all unlabeled data that belongs to the range (the level) are added to the expanded data. Thus, the model is expanded, that is, machine learning is used to label more data.

[0079] It can be seen that by checking the background whether the level of the label stratification reaches the preset threshold, it is determined whether to expand the label of all the data at this level as the data in the training set, thereby automatically labeling the data and adding more labeled data to help build and update the model, making the prediction results more accurate.

[0080] In one implementation of step S140, the training step to the step of adding the training set is repeated until all unlabeled data are labeled, specifically including repeating the processing of the aforementioned steps S110 to S130.

[0081] Furthermore, when repeating the training step, the prediction model is first rebuilt; when there is one or more new tags added, the prediction model adds predictions for the new one or more tags.

[0082] Furthermore, after the expansion, statistical indicators can be collected, such as whether a preset coverage rate is reached, or whether the coverage is 100%, etc. If the expected indicators (such as coverage rate, etc.) are met, the process ends; otherwise, the process continues to step S110 to step S130.

[0083] Take the repeated multiple rounds of autonomous annotation of text data as an example:

[0084] In round T, three data are randomly selected, and three labels A, B, and C are given to data 1, data 2, and data 3. Assume that data 1 is A, B, and C; data 2 is A and B; and data 3 is A and C (for example, provided to the front-end user for manual labeling). The model is trained with data 1, 2, and 3, and the other unlabeled data are input for prediction, and the label prediction probability is output. According to the label prediction probability, several strategies are sampled in turn to obtain sample data. The user manually checks the situation of these sample data, that is, verifies and labels them, and can also decide whether new labels need to be added (front-end). For example, if the unlabeled data 4 is extracted, the user checks that it needs to be labeled with A and C, and also needs to add a D, then data 4 can be labeled with A, C, and D. In the background, according to the predicted label prediction probability, the label stratification accuracy can be checked, mainly to determine whether the level of the label stratification can reach the threshold, such as 0.9. If it reaches it, all unlabeled data within the level range can be added to the training set, that is, the data expansion of the training model.

[0085] Before entering the next round, you can count the indicators to determine the ratio / coverage of the labeled data in the entire data that needs to be labeled. If it reaches 100%, the process ends, otherwise it enters the next round.

[0086] When entering the next round, such as T+1, the newly added data 4 to the training set is sent to the updated / established xBERT model to continue training and continue to predict the unlabeled data, repeating the processing of round T. During this process, the labels in training and labeling are A, B, C, and D.

[0087] It can be seen that updating the extended tags and annotating the data during annotation, this dynamic addition of tags and multi-tag annotation is in line with the development trend of data processing, adapts to more application scenarios and industry businesses, has strong data processing scalability, and helps to improve user experience.

[0088] Therefore, the multi-label annotation and modeling can be completed autonomously for data with uncertain labels. Users basically only need to annotate a certain number of labels (7k-8k) to obtain performance similar to that of full annotation (40,000-60,000) and manual annotation on most labels, which improves the annotation efficiency. In addition, it can also quickly verify whether the existing annotation standards can be implemented well (judged from the indicator data of the model expansion), etc.

[0089] [Example 2]

[0090] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the device implementation of the present invention is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0091] Here we will combine Figure 2 The main structural block diagram of an embodiment of the method of the present invention is illustrated in FIG. In this embodiment, the device at least includes:

[0092] A training prediction module B210 is used to train a prediction model using labeled data in a training set, and to predict unlabeled data using the prediction model;

[0093] The sampling and labeling module B220 is used to extract sample data for inspection based on the prediction results and decide whether to add new labels;

[0094] The extended labeling module B230 is used to check the accuracy of the predicted label stratification to determine whether all unlabeled label data that have reached a preset accuracy threshold are labeled and added to the training set, so as to continue model expansion (i.e., machine extended labeling);

[0095] Return to the repetition module B240, which is used to repeat the training steps to the step of adding to the training set until all unlabeled data are labeled.

[0096] In one embodiment of the training prediction module B210, the prediction model is trained by using labeled data in the training set, specifically including: the labeled data can be obtained by: randomly sampling one or more data to be processed, providing them to the front end, and having the user manually perform initial labeling of one or more labels on the sampled data.

[0097] Furthermore, when the user performs manual annotation, one or more existing tags recommended to the user may be displayed on the interactive interface of the application performing the annotation.

[0098] Further, the data initially labeled with one or more labels is added to the training set. And / or, the prediction model is specifically a model constructed by xBERT. Taking the multi-label labeling method with uncertain labels in the text data processing process as an example:

[0099] Preprocessing to obtain text data: preliminarily screen the text information and remove invalid text to obtain the preliminarily screened text; further perform word segmentation, stop word removal, statistics, etc. on the preliminarily screened text to obtain valid text data (valid word list). Even, cluster the texts, perform statistics to obtain the similarity between texts and representative texts, extract a certain amount of representative texts from the text collection according to clustering clusters, valid words, etc., and annotate them.

[0100] Perform initial annotation: randomly select one or more data to be processed from the text data. If the data to be processed has not been labeled before, it will be manually labeled by the user, which is the initial annotation. If some of the data have been labeled, it can directly become the training data of the model training set. Among them, the random sampling can pre-set a fixed number of sampled data, such as 200, 300, etc.

[0101] For example, in the case of manual annotation by users, the sentence (data to be processed) is "EU rejects German call to boycottBritish lamb.". At this time, the user enters the pre-defined three labels A, B, and C. When the user manually annotates the sentence, it is judged to belong to the three categories of A and B, and it is found that there should be a new label D, so the D label is continued to be entered, and the three labels A, B, and D are annotated. For other unlabeled data, it is not certain which label or labels should be annotated, so the labels are uncertain and intelligent automatic multi-label annotation is realized. The annotation labels can use various existing or self-developed rapid annotation platforms. The pre-defined labels such as A, B, and C can also be extracted from the historical labels of related projects. Each piece of text data uploaded by sampling is annotated, and the labeled data is exported.

[0102] Example 1:

[0103] An input file might look like this: import.json

[0104] {"text":"EU rejects German call to boycott British lamb.","meta":{"external_id":1}}

[0105] The exported file is as follows: output.json

[0106] {"doc_id":2023,"text":"EU rejects German call to boycott Britishlamb.","labels":["news"],["Trade"],["EU"]"}

[0107] Example 2:

[0108] Text category annotation:

[0109] Id: 1; What is the form of counter guarantee; label: 4, 7

[0110] Id: 2; What is the bank loan procedure; label: 4, 9

[0111] …

[0112] Example 3:

[0113] Keyword annotation of text data:

[0114] Id: 127; Compensation for vehicle damage on residential roads

[0115] Id: 129; Compensation for vehicle accident property damage liability disputes

[0116] …

[0117] Obtaining a training set: The labeled text data, including the text data initially labeled with one or more labels by the aforementioned user, is formed into a training set or added to the training set.

[0118] Model learning: training the established prediction model. The prediction model can be the algorithm part of the active learning platform that can perform automatic labeling. Specifically, xBERT can be used to build a model, two-way learning (NLP), and more accurate prediction. The training and modeling of the xBERT model for labeled data will not be described here. Preferably, the strategy that can be adopted for the labeled data used in model modeling: (1) If the labels are relatively balanced in the manually labeled data, then use manual inspection + machine expansion data; (2) If the labels are relatively unbalanced in the manually labeled data, then select the manually inspected data as the model training data. Preferably, the loss function for xBERT model training can adopt the cross entropy function loss, which consists of two parts: the first part is the manually labeled part, and the second part w mechine The part of the labeled data expanded for the machine, the two together serve as the loss function of the overall model. For example:

[0119]

[0120] Among them, the machine-expanded data is regarded as a positive example of the corresponding label as a whole, and its contribution to the loss function is adjustable; c It is a hyperparameter for adjusting the recall rate Recall and precision Precision. If it is greater than 1, it is biased towards Recall; n, z represent the total number of samples, c, represents the sample, y n,c represents the actual label, σ( n，c ), represents the predicted output. Furthermore, the second part only takes data, can cope with the situation where label uncertainty increases.

[0121] In one embodiment, the prediction model is used to predict the unlabeled data, specifically including: inputting the unlabeled data and the data with uncertain labels into the xBERT model, and predicting the prediction results corresponding to the output data, such as the label prediction probability, etc. For example, the label A prediction probability of text data 1 is 0.6, the label B prediction probability is 0.3, and the label C prediction probability is 0.1.

[0122] It can be seen that by combining manual labeling with machine learning model training and prediction, as much data as possible is predicted and labeled, and the labeled data is accumulated and added to the training set to train and update the model, multi-label labeling with uncertain labels is achieved. With a small amount of manual multi-label labeling, multi-label labeling of a large amount of data can be completed by autonomous learning, which improves labeling efficiency, reduces the burden of manual labeling, and reduces the cost of tedious work.

[0123] In one embodiment of the sampling and labeling module B220, extracting sample data for inspection based on the prediction results specifically includes: extracting sample data from the category corresponding to each label for inspection based on one or more label prediction probabilities output by the prediction model, and deciding whether to add new labels.

[0124] Further, the extracting of sample data specifically includes: extracting one or more sample data from the category corresponding to each label according to the label prediction probability and the predetermined sampling quantity. For example, stratified sampling is performed from all data according to the label prediction probability. Generally, there can be multiple strategies for extracting samples. In this strategy set, the optimal strategy is determined and recommended based on the feature analysis of the data currently labeled. For example, in the case of predicting labels for input data, it is mainly to determine the probability of whether to label a certain label. Preferably, the probability can be recommended to do stratified sampling strategy. The sampling process can refer to the existing sampling principles and will not be repeated here. Preferably, it includes but is not limited to one or more of the following sampling strategies: stratified sampling, vocabulary diversity sampling, random supplementary sampling, and machine-labeled data model score jitter sampling can be performed according to the label prediction probability. Further, the sampling strategy can be in the above order or can be changed in any combination. Among them, the vocabulary diversity sampling is adopted to consider the vocabulary division in all data and extract from the unlabeled data with a large difference from the vocabulary division of the labeled data. In addition, in order to deal with potential erroneous labels in the machine label expansion process, the machine label data model score jitter sampling is used, and the standard deviation of the historical prediction scores of the expanded labels of the data is calculated. If the standard deviation exceeds a specific threshold, the sampled data can be added for manual review.

[0125] Furthermore, in sampling, we can add a special tag "others" to target long-tail imbalanced tags, and exclude this special tag in vocabulary diversity sampling and random sampling processes to increase the probability of extracting potential positive examples.

[0126] In one embodiment, the inspection specifically includes: the user manually verifies the sample data. Furthermore, the extracted sample data includes or does not include data without labels, and the manual analysis verifies whether one or more labels corresponding to the data are accurate and correct, and then whether the labeling meets the requirements or satisfies various preset conditions. For example, the inspection process is a labeling process for the front-end user to determine whether the data is labeled, whether the labeling is incorrect, whether the labeling is complete / which labels are labeled, etc. based on the sampled data and labels provided on the interactive interface.

[0127] In one implementation, the user can manually check and inspect the extracted sample data to determine whether a new label needs to be added, and the new label added can be one or more.

[0128] It can be seen that automatic multi-label annotation is implemented for data with uncertain labels and its correctness is verified, which helps to improve the accuracy of subsequent multi-label annotation of data, thereby helping to update the constructed model to provide better predictions and autonomous annotations. The expansion of autonomous annotation based on machine learning can adapt to data annotation in more and wider application fields. In addition, sampling according to the label prediction probability and the coordination of label prediction probability can further improve the efficiency of intelligent annotation of complex and large data. By checking the sampled data, users can determine the accuracy and correctness of the model and decide whether to add one or more new labels.

[0129] In one embodiment of the extended labeling module B230, the accuracy of the predicted label stratification is checked to determine whether all unlabeled data that have reached a preset accuracy threshold are labeled and added to the training set. Specifically, it includes: checking the accuracy of the label stratification predicted by the model in the background. If the level reaches the preset accuracy value of 0.9, all unlabeled data that belongs to the range (the level) are added to the expanded data. Thus, the model is expanded, that is, machine learning labels more data.

[0130] It can be seen that by checking the background whether the level of the label stratification reaches the preset threshold, it is determined whether to expand the label of all the data at this level as the data in the training set, thereby automatically labeling the data and adding more labeled data to help build and update the model, making the prediction results more accurate.

[0131] In one embodiment of returning to repeat module B240, the training steps to the step of adding the training set are repeated until all unlabeled data are labeled, specifically including repeating the processing of the aforementioned modules B210 to B230.

[0132] Furthermore, when repeating the training step, the prediction model is first rebuilt; when there is one or more new tags added, the prediction model adds predictions for the new one or more tags.

[0133] Furthermore, after the expansion, statistical indicators can be collected, such as whether a preset coverage rate is achieved, or whether the coverage is 100%, etc. If the expected indicators (such as coverage rate, etc.) are met, the process ends; otherwise, the process continues to module B210 to module B230.

[0134] Take the repeated multiple rounds of autonomous annotation of text data as an example:

[0135] In round T, three data are randomly selected, and three labels A, B, and C are given to data 1, data 2, and data 3. Assume that data 1 is A, B, and C; data 2 is A and B; and data 3 is A and C (for example, provided to the front-end user for manual labeling). The model is trained with data 1, 2, and 3, and the other unlabeled data are input for prediction, and the label prediction probability is output. According to the label prediction probability, several strategies are sampled in turn to obtain sample data. The user manually checks the situation of these sample data, that is, verifies and labels them, and can also decide whether new labels need to be added (front-end). For example, if the unlabeled data 4 is extracted, the user checks that it needs to be labeled with A and C, and also needs to add a D, then data 4 can be labeled with A, C, and D. In the background, according to the predicted label prediction probability, the label stratification accuracy can be checked, mainly to determine whether the level of the label stratification can reach the threshold, such as 0.9. If it reaches it, all unlabeled data within the level range can be added to the training set, that is, the data expansion of the training model.

[0136] Before entering the next round, you can count the indicators to determine the ratio / coverage of the labeled data in the entire data that needs to be labeled. If it reaches 100%, the process ends, otherwise it enters the next round.

[0137] When entering the next round, such as T+1, the newly added data 4 to the training set is sent to the updated / established xBERT model to continue training and continue to predict the unlabeled data, repeating the processing of round T. During this process, the labels in training and labeling are A, B, C, and D.

[0138] It can be seen that updating the extended tags and annotating the data during annotation, this dynamic addition of tags and multi-tag annotation is in line with the development trend of data processing, adapts to more application scenarios and industry businesses, has strong data processing scalability, and helps to improve user experience.

[0139] Therefore, the multi-label annotation and modeling can be completed autonomously for data with uncertain labels. Users basically only need to annotate a certain number of labels (7k-8k) to obtain performance similar to that of full annotation (40,000-60,000) and manual annotation on most labels, which improves the annotation efficiency. In addition, it can also quickly verify whether the existing annotation standards can be implemented well (judged from the indicator data of the model expansion), etc.

[0140] [Example 3]

[0141] The following describes an overall application example, and further illustrates the implementation process of the present invention in combination with Examples 1 and 2. Figure 6 The main process of a specific example is shown as well as Figure 7The front-end user annotation interface is displayed. Create an annotation project and start running the annotation. The front-end user performs manual annotation through the interface, while the modeling training and prediction are performed in the background to automatically expand the machine annotation to achieve automatic annotation of more data. The goal of this embodiment is mainly to perform a corresponding modeling process for the situation where the initial annotation label is uncertain in the annotation process, so as to achieve the annotation of more or even all data.

[0142] Core processes include:

[0143] 1. The user provides labeled data. First, determine whether the input data is partially labeled. If the input has no labeled data, go to the following 2 sampling data. If there is partially labeled data, go to the following 3 model learning.

[0144] 2. Randomly sample an initial fixed number of data and allow users to start multi-label annotation. Figure 7 , showing that there are multiple tags, some are recommended by the machine, and some are new tags added by the user according to their needs, etc., a piece of text data (sentence) is labeled, which may be one or more tags.

[0145] 3. Use the xBERT model to train and model the labeled data (model learning); use the model to perform model prediction, and perform stratified sampling, vocabulary diversity sampling, random sampling, and even machine-labeled data model score jitter sampling according to label prediction probability.

[0146] Among them, there are currently two strategies for the labeled data used in model building: C1. If the labels in the manually labeled data are relatively balanced, use manual inspection + machine expansion data; C2. If the labels in the manually labeled data are relatively unbalanced, intelligently select manually inspected data as model training data.

[0147] The loss function (cross entropy loss function) of the xBert model training is as follows:

[0148]

[0149] Part I - ∑ n，c [p cyn,c ·logσ(x n,c )+(1-y n,c )·log(1-σ(x n,c ))] is manually annotated data;

[0150] Part 2 This is the machine-expanded label data, and only the corresponding data of yz,c=1 is taken. This is to deal with the situation where label uncertainty increases.

[0151] Here, the machine-expanded data is treated as a positive example of the corresponding label as a whole, and its contribution to the loss function is adjustable.

[0152] pc is a hyperparameter for adjusting Recall and Precision. If it is greater than 1, it is biased towards Recall.

[0153] The specific training principle of the BERT model will not be elaborated here. For the symbolic explanation, please refer to the principle of the cross entropy loss function and the description of Examples 1 and 2.

[0154] In the core sampling strategy, (a) in response to the long-tail imbalanced labels, the special label "others" is added, and this special label is excluded in the vocabulary diversity sampling and random sampling process to increase the probability of extracting potential positive examples. (b) The vocabulary diversity sampling it uses: it considers the vocabulary distribution in all data and extracts from unlabeled data that has a large difference in vocabulary distribution from the labeled data. (c) In order to deal with potential erroneous labels in the machine expansion process, the machine-labeled data model score jitter sampling is used, and the standard deviation of the historical prediction score of the expanded label of the data is calculated. If the standard deviation exceeds a specific threshold, the sampled data is added and submitted to manual review, etc.

[0155] The labeling process achieves the following effects: users basically only need to label a certain number of labels (7k-8k) to obtain performance similar to that of full labeling (40,000-60,000) on most labels, which improves labeling efficiency; it can quickly verify whether the existing labeling standards can be implemented well (judged from the indicator data of the model expansion). Therefore, in the front-end labeling interface, when users, i.e. labelers, see a piece of data, they will label all the relevant labels they can think of.

[0156] [Example 4]

[0157] Figure 3 It is a schematic block diagram of the structure of an electronic device according to an embodiment of the present invention, wherein the electronic device includes a processor and a memory, wherein the memory is used to store a computer executable program, and when the computer program is executed by the processor, the processor executes the embodiment steps of the method involved in the aforementioned embodiments 1 and 3.

[0158] like Figure 3 As shown, the electronic device is presented in the form of a general-purpose computing device. The processor may be one or more and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0159] The memory stores a computer executable program, usually a machine-readable code, which can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least part of the steps in the method.

[0160] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

[0161] Optionally, in this embodiment, the electronic device further includes an I / O interface, which is used for the electronic device to exchange data with external devices. The I / O interface can represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0162] More specifically, see Figure 4 1 is a block diagram of a more specific example of the electronic device described in the embodiment shown. The electronic device 200 of the exemplary embodiment is in the form of a general data processing device. The components of the electronic device 200 may include, but are not limited to: at least one processing unit 210, at least one storage unit 220, a bus 230 connecting different system components (including the storage unit 220 and the processing unit 210), a display unit 240, etc.

[0163] The storage unit 220 stores a computer-readable program, which may be a source program or a code of a read-only program. The program may be executed by the processing unit 210, so that the processing unit 210 performs the steps of various embodiments of the present invention. For example, the processing unit 210 may perform the various steps of the methods of the aforementioned embodiments 2 to 5.

[0164] The storage unit 220 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 2201 and / or a cache storage unit 2202, and may further include a read-only storage unit (ROM) 2203. The storage unit 220 may also include a program / utility 2204 having a set (at least one) of program modules 2205, such program modules 2205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include the implementation of a network environment.

[0165] Bus 230 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0166] The electronic device 200 may also communicate with one or more external devices 300 (e.g., keyboards, displays, network devices, Bluetooth devices, etc.) so that a user can interact with the electronic device 200 via these external devices 300, and / or so that the electronic device 200 can communicate with one or more other data processing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 250, and may also be performed through a network adapter 260 with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet). The network adapter 260 may communicate with other modules of the electronic device 200 via the bus 230. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in the electronic device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0167] It should be understood that Figure 3 , 4 The electronic device shown is only an example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.

[0168] [Example 5]

[0169] Figure 5 Schematic diagram of a computer readable recording medium according to an embodiment of the present invention. Figure 5 As shown, a computer executable program is stored in a computer-readable recording medium, and when the computer executable program is executed, the above-mentioned method for providing a tutoring strategy based on a question of the present invention is implemented. The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0170] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0171] Through the above description of the implementation mode, it is easy for those skilled in the art to understand that the present invention can be implemented by hardware capable of executing a specific computer program, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. contained in the system. The present invention can also be implemented by a vehicle comprising at least a part of the above system or components. The present invention can also be implemented by computer software that executes the method of the present invention, such as control software executed by a microprocessor, an electronic control unit, a client, a server, etc. on the locomotive side. However, it should be noted that the computer software that executes the method of the present invention is not limited to being executed by one or a specific hardware entity, and it can also be implemented in a distributed manner by unspecified specific hardware, such as some method steps executed by the computer program can be executed on the locomotive side, and another part can be executed in a mobile terminal or a smart helmet, etc. For computer software, the software product can be stored in a computer-readable storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), or it can be distributed and stored on the network, as long as it enables the electronic device to execute the method according to the present invention.

[0172] Through the above description of the implementation modes, those skilled in the art can easily understand that the exemplary embodiments described in the present invention can be implemented by software, or by combining software with necessary hardware.

[0173] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the present invention is not inherently related to any specific computer, virtual device or electronic device, and various general devices can also implement the present invention. The above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A label marking method, characterized in that: include: The prediction model is trained by the labeled data in the training set, and the unlabeled data is predicted by the prediction model; the prediction model is specifically a model constructed by xBERT, and the loss function of the prediction model training includes a cross-entropy loss function calculated from two parts: the user's manually labeled data and the model's extended labeled data; wherein the labeled data is obtained by randomly sampling an initial fixed number of data, and the user manually annotates the sampled data with one or more labels, and the data initially annotated with one or more labels is added to the training set; wherein the labeled data includes text data, voice data or image data, wherein the text data needs to be initially annotated through preliminary screening, text clustering, and then extracting representative text from the clustered text set; Extracting sample data for inspection according to the prediction result and deciding whether to add new labels include: extracting sample data from the category corresponding to each label for inspection according to one or more label prediction probabilities output by the prediction model; the extracting sample data specifically includes at least one or more sampling of stratified sampling, vocabulary diversity sampling, random supplementary sampling, and machine label data model score jitter sampling according to the label prediction probability; the inspection specifically includes the user manually verifying the sample data to mark the sample data and determining whether to add one or more new labels according to the verification; Checking the stratification of labels predicted by the prediction model to determine whether it is necessary to label the data without labels and add them to the training set, including: checking the accuracy of the predicted label stratification, if the stratification level reaches a preset accuracy threshold, all the data without labels within the range of the stratification are labeled and added to the training set to expand the labeled data; Repeat the training steps to the step of adding to the training set until the expected labeling conditions are met.

2. The label marking method according to claim 1, characterized in that: Also includes: Before training the prediction model, first determine whether part of the data input to the prediction model has been labeled; If yes, directly input it into the prediction model for model training; If otherwise, it means that no data is labeled, it is necessary to randomly sample an initial fixed number of data from the data input into the prediction model and perform the initial labeling.

3. The label marking method according to claim 1, characterized in that: Also includes: To address the long-tail imbalanced labels, we added the special label "others", which is excluded from the vocabulary diversity sampling and random sampling processes to increase the probability of extracting potential positive examples. and / or, The vocabulary diversity sampling refers to the vocabulary division of all data, which is extracted from the unlabeled data that differs greatly from the vocabulary division of the labeled data; and / or, The machine-labeled data model score jitter sampling is used, and the standard deviation of the historical prediction scores of the extended annotation labels of the unlabeled data is counted. If the standard deviation exceeds a specific threshold, it is used as the extracted sample data to deal with potential erroneous labels in the extended annotation.

4. The label marking method according to any one of claims 1 to 3, characterized in that: Repeat the training steps to the step of adding to the training set until the expected labeling conditions are met, including: Before repeating the training step, count the labeled data to determine whether the expected coverage is achieved; If yes, then end the extended annotation data; If otherwise, start repeating the cycle of training the prediction model; And, when there are one or more new labels added, the prediction model adds predictions for the one or more new labels during training and prediction.

5. The label marking method according to any one of claims 1 to 3, characterized in that: Also includes: When training the prediction model, if the labels in the manually annotated data are relatively balanced, the manually annotated data and the data annotated by the model are used; If the labels in the manually annotated data are unbalanced, the manually inspected data is selected as the model training data.

6. The label marking method according to claim 4, characterized in that: Also includes: When training the prediction model, if the labels in the manually annotated data are relatively balanced, the manually annotated data and the data annotated by the model are used; If the labels in the manually annotated data are unbalanced, the manually inspected data is selected as the model training data.

7. A label marking device, characterized in that: include: A training prediction module is used to train a prediction model through labeled data in a training set, and to predict unlabeled data through the prediction model; the prediction model is specifically a model constructed by xBERT, and the loss function during the training of the prediction model includes a cross-entropy loss function calculated from two parts: the user's manually labeled data and the model's extended labeled data; wherein the labeled data is obtained by randomly sampling an initial fixed number of data, and the user manually annotates the sampled data with one or more labels, and the data initially annotated with one or more labels is added to the training set; wherein the labeled data includes text data, voice data or image data, wherein the text data needs to be initially annotated through preliminary screening, text clustering, and then extracting representative text from the clustered text set; A sampling inspection module is used to extract sample data for inspection according to the prediction results and decide whether to add new labels, including: extracting sample data from the category corresponding to each label for inspection according to one or more label prediction probabilities output by the prediction model; the extracting sample data specifically includes at least one or more sampling of stratified sampling, vocabulary diversity sampling, random supplementary sampling, and machine label data model score jitter sampling according to the label prediction probability; the inspection specifically includes the user manually verifying the sample data to mark the sample data and determining whether to add one or more new labels based on the verification; The extended labeling module is used to check the stratification of the labels predicted by the prediction model to determine whether it is necessary to label the data without labels and add them to the training set, including: checking the accuracy of the predicted label stratification, if the stratification level reaches a preset accuracy threshold, all the data without labels within the range of the stratification level are labeled and added to the training set to expand the labeled data; Return the repeat module, which is used to repeat the training steps to the step of adding to the training set until the expected labeling conditions are met.

8. A computer-readable medium storing a computer-executable program, wherein when the computer-executable program is executed, the method according to any one of claims 1 to 6 is implemented.

Images