Text sample labeling method and device

Abstract

The embodiment of the invention provides a text sample labeling method and device, and the method comprises the steps: obtaining a to-be-labeled first text sample from a sample pool; and carrying out labeling processing on the first text sample by utilizing the model to obtain a prediction label of the first text sample. And through a target classifier, based on the first text sample and the prediction label thereof, determining a sample category to which the first text sample belongs, the sample category indicating the degree of certainty of the prediction label. And executing target processing corresponding to the sample category to which the first text sample belongs for the first text sample.

Description

Technical Field

[0001] The embodiments of this specification belong to the field of machine learning, and more particularly to a method and device for labeling text samples. Background Art

[0002] Machine learning has been widely used in various fields, such as natural language processing (NLP), and has achieved good performance. However, the performance of machine learning models depends largely on the quantity and quality of training samples. Insufficient data of training samples will lead to poor generalization ability of the model. For example, in text classification tasks, there is often a lack of a large number of annotated text samples, which leads to low prediction accuracy of text classification models and is difficult to meet actual needs. However, the annotation of text samples requires a lot of manpower input. Therefore, it is necessary to provide a more effective text sample annotation solution. Summary of the invention

[0003] The purpose of the present invention is to provide a method and device for annotating text samples, which can greatly improve the efficiency and accuracy of text sample annotation.

[0004] A first aspect of the present specification provides a method for annotating a text sample, comprising:

[0005] Acquire a first text sample to be annotated from a sample pool;

[0006] Using the model to label the first text sample to obtain a predicted label for the first text sample;

[0007] Determining, by a target classifier, based on the first text sample and its predicted label, a sample category to which the first text sample belongs, which indicates a degree of confidence of the predicted label;

[0008] Target processing corresponding to the sample category to which the first text sample belongs is performed on the first text sample.

[0009] A second aspect of the present specification provides a text sample annotation device, comprising:

[0010] An acquisition unit, used for acquiring a first text sample to be annotated from a sample pool;

[0011] A labeling unit, used to label the first text sample using the model to obtain a predicted label of the first text sample;

[0012] a determination unit, configured to determine, by a target classifier, based on the first text sample and its predicted label, a sample category to which the first text sample belongs, which indicates a degree of confidence of the predicted label;

[0013] An execution unit is used to perform target processing corresponding to the sample category to which the first text sample belongs.

[0014] A third aspect of the present specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method described in the first aspect.

[0015] A fourth aspect of the present specification provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method of the first aspect is implemented.

[0016] The fifth aspect of this specification provides a computer program product, including a computer program / instruction, which implements the steps of the method of the first aspect when executed by a processor.

[0017] The text sample annotation method provided by one or more embodiments of this specification first annotates the text sample to be annotated using a model, and then determines the sample category of the text sample based on the annotated text sample and the classifier, which indicates the degree of confidence of the annotation result. Finally, the text sample is post-processed differently according to the different degrees of confidence of the annotation result. It can be seen that this scheme annotates text samples by combining model annotation with active learning, thereby greatly improving the annotation efficiency and accuracy of text samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 is a flow chart of a method for marking text samples in an embodiment of this specification;

[0020] Figure 2 is a schematic diagram of a method for marking text samples in an embodiment of this specification;

[0021] Figure 3 It is a schematic diagram of a device for marking text samples in one embodiment of this specification. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0023] The inventor of this solution found that there are often three major pain points in labeling scenarios:

[0024] (1) High label complexity: The complexity of modern business problems often requires multi-level, fine-grained labeling, which makes the labeling system very complex. In addition, complex labeling systems are often accompanied by frequent changes in labeling rules, which increases the difficulty and error rate of labeling. At the same time, due to the high complexity of labels, each category often requires more samples to fully learn.

[0025] (2) Many business custom rules: Various business scenarios are often accompanied by unique requirements, which leads to the generation of custom rules. The diversity of custom rules requires labelers to not only be familiar with conventional labeling rules, but also be able to flexibly adapt to various special rules, which increases the difficulty of labeling. For example, in the SMS classification scenario, the logistics SMS sent by e-commerce merchants are classified as logistics industry or e-commerce industry.

[0026] (3) Poor consistency in manual annotations, low Kappa: The Kappa coefficient is a statistic that evaluates the consistency of annotations between multiple annotators. A low value of this indicator usually indicates disagreements between the annotation teams, which directly undermines the overall consistency and reliability of the dataset. Inconsistent annotations can cause the model to learn inaccurate or contradictory signals, which in turn affects its performance.

[0027] In order to solve the above three pain points, the following two text sample annotation solutions are proposed:

[0028] The first is the annotation method based on the Human In The Loop annotation framework; this annotation method is a human-machine collaborative method that combines human input with the operation of machine algorithms to optimize the performance of task execution. The core idea is to embed human review or intervention steps at the key decision points of the automated system. It mainly includes the following main steps: (1) Automated processing: Use existing machine learning models or algorithms to perform preliminary processing of data. This may include tasks such as text classification and entity recognition. (2) Human review: Introduce human review for situations where the algorithm is uncertain or for key decision nodes. Human reviewers will check the machine output and verify, correct or supplement it to ensure the accuracy of the results. (3) Feedback loop: The results of human review are used to train or adjust the machine learning model. Through this continuous feedback loop, the performance of the model gradually improves over time. (4) Iterative optimization: By repeating this process periodically, the accuracy and efficiency of the model are continuously optimized while reducing the need for human intervention.

[0029] The disadvantages of this solution are: (1) Time-consuming and costly: It all depends on manual operations. For large-scale data sets, the labeling process is time-consuming and costly. (2) Consistency issues: Manual labeling may be subjective, and the labeling results between different labelers may vary, affecting the consistency of the data.

[0030] The second method is the labeling method based on the large model: this method can be performed when there are no or only a small number of labeled text samples. It mainly activates and uses the knowledge learned by the large model through "prompts" and in-context learning to label text samples.

[0031] The disadvantages of this solution are: Annotation quality problem: Due to the limited reference context of large models and the lack of optimization of the model for specific tasks, the generated annotation labels are of poor quality.

[0032] To this end, in the embodiments of this specification, an improved scheme is proposed, which first uses a model to annotate the text sample to be annotated, and then determines the sample category of the text sample based on the annotated text sample and the classifier, which indicates the degree of confidence of the annotation result. Finally, according to the different degrees of confidence of the annotation results, the text samples are screened, for example, the samples with the least number and the most valuable are screened for manual annotation. It can be seen that this scheme annotates text samples by combining model annotation with active learning, thereby greatly improving the annotation efficiency and accuracy of text samples.

[0033] Figure 1This is a flow chart of a method for labeling text samples in an embodiment of this specification. The method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. It should be noted that the method includes multiple rounds of iterations. Figure 1 The method steps included in the tth (t is a positive integer) iteration are shown. It can be understood that by repeatedly executing the steps shown therein, multiple rounds of annotation of text samples can be achieved, and then the text sample after the last round of annotation is used as the final text sample. Figure 1 As shown, the method may include the following steps:

[0034] Step S102, obtaining a text sample s1 to be annotated from a sample pool.

[0035] The text sample s1 to be labeled here may refer to a text sample with an incorrect category label or a text sample without a category label.

[0036] The above-mentioned text sample s1 may include several text features, including but not limited to text content, text metadata (such as author, release time, language, length, etc.), and may also include words extracted from a sentiment dictionary.

[0037] Step S104: use the model to label the text sample s1 to obtain a predicted label for the text sample s1.

[0038] The model here may refer to a large model or other models that can be used to generate sample labels. In the following description, the model here is taken as an example to illustrate.

[0039] The predicted label can be selected from a predetermined k categories, where k is a positive integer. Typically, in the scenario of sentiment polarity classification, the k categories include positive and negative, and the two categories can be represented by 0 and 1, respectively. In some other scenarios, specifically in the SMS text classification scenario, since SMS texts usually need to be classified into multiple industries, the above k categories can be multiple industries.

[0040] Specifically, the large model can be used to label the text sample s1 based on contextual learning examples and / or labeling rules.

[0041] When performing annotation processing based on contextual learning examples, the annotation sample set can be queried based on text sample s1 to obtain a text sample s2 that matches the text sample s1, where each annotated text sample in the annotated sample set has a pre-annotated category label. Text sample s1, text sample s2 and their category labels are input into the large model to obtain a predicted label for text sample s1.

[0042] The i-th annotated text sample in the above annotated sample set is recorded as (x i ,y i ), where x i Represents the text features of the i-th annotated text sample, which may also include text content and text metadata, etc. i It represents the category label of the i-th text sample, which can be annotated manually. The definition of the category label can refer to the above-mentioned prediction label. That is, the category label can be positive or negative in the sentiment polarity classification scenario, or a certain industry among multiple industries, etc.

[0043] The selection process of the above-mentioned text sample s2 can be: first, based on the text features of the text sample s1, use the representation model to determine the text vector of the text sample s1, and based on the text features of each annotated text sample, use the representation model to determine the text vector of each annotated text sample. Afterwards, based on the text vector of the text sample s1 and the text vectors of each annotated text sample, calculate the similarity between the text sample s1 and each annotated text sample, and select each annotated text sample whose corresponding similarity is greater than a predetermined threshold as the above-mentioned text sample s2. Alternatively, the annotated text samples can be sorted from large to small according to the corresponding similarity, and the top m annotated text samples can be selected as the above-mentioned text sample s2. That is, the number of the above-mentioned text samples s2 can be multiple.

[0044] Among them, the above-mentioned representation model can be implemented as a Bert model, a Transformer encoder, etc.

[0045] It should be understood that in this annotation processing method, the large model actually generates the predicted label of the text sample s1 by learning the association between the text sample s2 and its category label.

[0046] In addition, when performing annotation processing based on the annotation rule, the rule library can be queried based on the text sample s1 to obtain the target annotation rule that matches it. The text sample s1 and the target annotation rule are input into the large model to obtain the predicted label of the text sample s1.

[0047] The rule base stores a plurality of labeling rules, which may be generated manually or by using a large model.

[0048] Among them, the process of using the big model to generate labeling rules can be: input each labeled text sample and its category label in the labeled sample set into the big model, and instruct the big model through prompt text to generate each labeling rule by learning the association between each labeled text sample and its category label, and then form a rule base based on each labeling rule.

[0049] The annotation rules in the rule base exist in the form of natural language, which can also be regarded as a kind of text, so the query process of the above-mentioned target annotation rules is similar to the query process of the above-mentioned text sample s2, that is, first based on the text features of each annotation rule, use the characterization model to determine the text vector of each annotation rule. Then, based on the text vector of each annotation rule and the text vector of the text sample s1, calculate the similarity between the text sample s1 and each annotation rule, and select each annotation rule whose corresponding similarity is greater than a predetermined threshold as the above-mentioned target annotation rule. Alternatively, the annotation rules can be sorted from large to small according to the corresponding similarity, and the top m annotation rules can be selected as the above-mentioned target annotation rules. That is, the number of the above-mentioned target annotation rules can be multiple.

[0050] It should be noted that some additional business logic and domain knowledge can usually be introduced through labeling rules, so that when the large model labels text samples based on the labeling rules, it can reduce repeated errors and optimize the labeling process.

[0051] Finally, when labeling is performed based on contextual learning examples and labeling rules at the same time, text sample s1, text sample s2 and their category labels as well as target labeling rules can be input into the large model to obtain the predicted label of text sample s2.

[0052] Of course, in practice, in order to improve the accuracy and reliability of the big model annotation, when the big model annotates text samples in a specific field (such as medicine or law, etc.), the knowledge of the specific field can also be input into the big model.

[0053] Similar to the labeling process of text sample s1, a large model can be used to label each text sample to be labeled in the sample pool, so as to obtain a predicted label for each text sample to be labeled.

[0054] Step S106, determining the sample category to which the text sample s1 belongs based on the text sample s1 and its predicted label through the target classifier, which indicates the degree of confidence of the predicted label.

[0055] The sample categories here may include: high-confidence samples, medium-confidence samples and low-confidence samples. Of course, in practice, the sample categories may also include only high-confidence samples and low-confidence samples or other numbers, and this specification does not limit this.

[0056] The above-mentioned target classifiers may include logistic regression models, support vector machines, probabilistic graphical models, Bert-based classifiers, and the like.

[0057] Specifically, a target classifier may be trained based on each text sample to be labeled and its predicted label, and then the sample category to which each text sample to be labeled belongs may be determined according to the prediction results output by the target classifier for each text sample to be labeled during or after the training target classification process.

[0058] More specifically, the text sample s1 can be input into the target classifier to obtain the corresponding prediction result. Then, based on the probability and prediction label corresponding to the prediction label in the prediction result, the prediction loss corresponding to the text sample s1 is calculated using the cross entropy loss function. Finally, based on the prediction loss corresponding to each text sample to be labeled, the back propagation method is used to calculate the update gradient of the parameters corresponding to the target classifier, and then the parameters of the target classifier are updated based on the update gradient to obtain the trained target classifier.

[0059] Among them, the above cross entropy loss function is shown in Formula 1:

[0060] L CE =-∑ i y i log(p i ) (Formula 1)

[0061] If i = label, then y i =1, otherwise y i =0. Here, label is the predicted label corresponding to the input sample, p i The probability that the input sample belongs to category i predicted by the target classifier.

[0062] In other implementations, target text samples corresponding to the sample category of high-confidence samples in the previous round (i.e., the t-1th round) of iteration can also be selected from the text samples to be annotated, and then the target classifier is trained based on the target text samples and their predicted labels, which is not limited in this specification. It should be understood that since the predicted labels of high-confidence samples have a higher accuracy rate, this training method can improve the accuracy of the trained target classifier.

[0063] It should be noted that when training the target classifier based on the text samples to be labeled, the target classifier can also be trained based on the labeled text samples and their category labels in the labeled sample set, that is, a training text set for the target classifier is formed based on the text samples to be labeled and their predicted labels, as well as the labeled text samples and their category labels.

[0064] It should also be noted that this solution can determine the sample category indicating the degree of confidence of the label to which the text sample s1 belongs based on the prediction result output by the target classifier for the text sample s1 during the training of the target classifier. Alternatively, after the target classifier is trained, the text sample s1 can be input into the trained target classifier to obtain the corresponding prediction result, and based on the prediction result, the sample category indicating the degree of confidence of the label to which the text sample s1 belongs can be determined.

[0065] It should be understood that when the target classifier is a classifier for classifying k categories, the two prediction results both include k probabilities corresponding to the k categories.

[0066] In one embodiment, the sample category to which the text sample s1 belongs may be determined based on the confidence level (ie, the maximum probability among k probabilities).

[0067] Specifically, when the maximum probability among the above k probabilities falls within the predefined confidence interval, the text sample s1 is determined to be a medium-confidence sample; when the above maximum probability is less than the lower limit value of the confidence interval, the text sample s1 is determined to be a low-confidence sample; when the above maximum probability is greater than the upper limit value of the confidence interval, the text sample s1 is determined to be a high-confidence sample.

[0068] In another embodiment, the sample category to which the text sample s1 belongs may be determined based on the minimum margin (ie, the gap between two categories with the smallest probability difference).

[0069] Specifically, the above k probabilities can be subtracted from each other to obtain multiple difference values, and then the minimum difference value is selected from the multiple difference values, and when the minimum difference value falls within a predefined difference interval, it is determined that the text sample s1 belongs to a medium-confidence sample; when the minimum difference value is less than the lower limit value of the difference interval, it is determined that the text sample s1 belongs to a low-confidence sample; when the minimum difference value is greater than the upper limit value of the difference interval, it is determined that the text sample s1 belongs to a high-confidence sample.

[0070] In yet another embodiment, the sample category to which the text sample s1 belongs may be determined based on information entropy.

[0071] Specifically, the first entropy value can be calculated based on k probabilities. When the first entropy value falls within a predefined entropy value interval, it is determined that the text sample s1 belongs to a medium-confidence sample; when the first entropy value is less than the lower limit of the entropy value interval, it is determined that the text sample s1 belongs to a high-confidence sample; when the first entropy value is greater than the upper limit of the entropy value interval, it is determined that the text sample s1 belongs to a low-confidence sample.

[0072] In an example, the calculation formula of the first entropy value may be as follows:

[0073]

[0074] Among them, p i is the i-th probability among the k probabilities.

[0075] Step S108, executing target processing corresponding to the sample category to which the text sample s1 belongs.

[0076] The target processing here can also be understood as the post-processing of the labeling process.

[0077] Specifically, when the text sample s1 is a medium-confidence sample, the text sample s1 can be returned to the sample pool to be re-labeled using the large model; when the text sample s1 is a low-confidence sample, a category label is added to the text sample s1 manually, and the text sample s1 with the added category label is added to the labeled sample set; when the text sample s1 is a high-confidence sample, the corresponding predicted label is used as its category label, and the text sample s1 with the added category label is added to the labeled sample set. The target classifier can then be trained using the updated labeled sample set.

[0078] It can be seen that in this solution, when text sample s1 is not a high-confidence sample, the predicted label of text sample s1 will be recalibrated using the large model, or it will be annotated manually, which is equivalent to introducing new information to the training sample set (including text samples in the sample pool and annotated text samples in the annotated sample set), so that the overall distribution of the training sample set has changed. Therefore, the target classifier can continue to be trained based on the training text set in the next round. In addition, this ensures that the model training uses higher quality data, while also reducing the burden of manual review and avoiding the inefficiency of manual operation.

[0079] Similarly, other text samples to be annotated can be subjected to corresponding target processing.

[0080] It should be understood that in this scheme, continuously selecting high-confidence samples from the sample pool and adding low-confidence samples that have been manually labeled to the labeled sample set actually adopts an active learning strategy that can maximize the performance improvement of the model with minimal labeling effort.

[0081] So far, a round of standard processing of each text sample in the sample pool has been completed. It should be understood that by cyclically executing steps S102 to S108, multiple rounds of annotation processing of each text sample in the sample pool can be achieved. The termination condition of the cyclic process can include any one of the following:

[0082] The number of low-confidence samples in the sample pool is 0;

[0083] The performance index of the target classifier reaches a first threshold;

[0084] The improvement of the performance indicator of the target classifier is less than the second threshold.

[0085] The above performance indicators may include but are not limited to accuracy, precision, recall, F1 score, etc.

[0086] In addition, it should be understood that after the above-mentioned cycle process is completed, the target classifier obtained in the last round of cycle training can be used as the classifier finally used, or the classifier finally used can be retrained based on the text samples after the last round of annotation processing in the sample pool and the annotated text samples in the annotated sample set. This specification does not limit this.

[0087] Figure 2 It is a schematic diagram of a method for labeling text samples in one embodiment of this specification. Figure 2 In the example, the rule module is used to generate annotation rules based on the annotated text samples manually or using a large model. The large model annotation module is used to annotate the text samples to be annotated using a large model. The weakly supervised learning module is used to train the target classifier through weakly supervised learning. The sample classification module is used to classify the text samples to be annotated in the sample pool into different sample categories.

[0088] Specifically, for the text samples to be annotated in the sample set to be annotated, similar annotation rules can be queried from the rule module, and similar annotated texts can be queried from the annotated sample set (also called the annotated sample set). Then, similar annotation rules and similar annotated texts can be input into the large model annotation module to annotate the text samples to be annotated through the large model to obtain the predicted labels of the text samples to be annotated. After that, the text samples to be annotated and their predicted labels, as well as the annotated text samples and their category labels can be input into the weakly supervised learning module to train the target text classifier based on the two types of text samples, and the prediction results output by the target classifier for the text samples to be annotated during or after the training of the target classifier are input into the sample classification module, so that the sample classification module divides the text samples to be annotated into high-confidence samples, medium-confidence samples and low-confidence samples based on the prediction results.

[0089] For the high-confidence sample, it can be added to the labeled sample set, and it can be fed back to the rule module to generate new labeling rules. For the medium-confidence sample, it can be returned to the sample set to be labeled to be re-labeled. For the low-confidence sample, you can manually add a category label to it, and after adding the category label, add it to the labeled sample set.

[0090] In summary, this solution adopts a combination of active learning and model annotation. Specifically, the model is used to generate annotation samples preliminarily, and then samples with large amounts of information are selected through active learning for review and confirmation by manual experts, which helps to quickly improve the quality and quantity of the annotation sample set and reduce the cost of manual annotation. Through the strategy of cyclic iteration of manual annotation and model annotation, the quality of the annotation sample set and the accuracy of the annotation rules are continuously improved, and the consistency of the overall annotation is gradually improved. In addition, the rule module introduced in this solution uses the method of domain experts to formulate or summarize manual annotation samples to generate annotation rules. The annotation rules exist in the form of natural language to guide subsequent annotations, and through the annotation rules, business logic and domain knowledge can be effectively combined. The domain knowledge facilitates the definition of boundary conditions and avoids the recurrence of errors, thereby improving the efficiency of manual annotation. Finally, this solution combines the experience and knowledge of domain experts and the patterns learned from data to form a dual optimization method that integrates expert insights and data-driven insights. This strategy can improve the generalization ability of the model and its ability to handle boundary conditions, especially in complex problem situations.

[0091] The advantages of this solution are further described below:

[0092] 1. This solution is applicable to various text annotation tasks, such as sentiment analysis, text classification, and entity recognition. It is also applicable to text annotation tasks in various languages. It also supports customization of various labels and boundary rules to facilitate the repair of bad cases. 2. This solution supports different types of weakly supervised classifiers, such as logistic regression, language models (such as Bert, etc.), probabilistic graphical models, etc., which can be flexibly selected. 3. This solution improves efficiency by more than 50% compared to manual annotation while keeping the performance indicators of the test set basically the same.

[0093] Corresponding to the above-mentioned text sample annotation method, an embodiment of the present specification also provides a text sample annotation device, such as Figure 3 As shown, the device may include:

[0094] The acquisition unit 302 is configured to acquire a first text sample to be annotated from a sample pool.

[0095] The labeling unit 304 is used to label the first text sample using the model to obtain a predicted label of the first text sample.

[0096] A determination unit 306, configured to determine, through a target classifier, based on the first text sample and its predicted label, a sample category to which the first text sample belongs, which indicates a degree of confidence of the predicted label;

[0097] The execution unit 308 is used to perform target processing corresponding to the sample category to which the first text sample belongs.

[0098] In one embodiment, the determining unit 306 is specifically configured to:

[0099] Based on the first text sample and its predicted label, training a target classifier;

[0100] The sample category to which the first text sample belongs is determined according to a first prediction result output by the target classifier for the first text sample during or after the process of training the target classifier.

[0101] In one embodiment, the labeling unit 304 includes:

[0102] A query submodule 3042 is used to query a labeled sample set based on the first text sample to obtain a second text sample that matches the first text sample, wherein each labeled text sample in the labeled sample set has a pre-labeled category label;

[0103] The input submodule 3044 is used to input the first text sample, the second text sample and their category labels into the model to obtain a predicted label of the first text sample.

[0104] In another embodiment, the query submodule 3042 is further used to query the rule library based on the first text sample to obtain a target annotation rule that matches it;

[0105] The input submodule 3044 is further used to input the first text sample and the target labeling rule into the model to obtain a predicted label of the first text sample.

[0106] In one embodiment, the apparatus further comprises:

[0107] An input unit 310 is used to input each annotated text sample and its category label in the annotated sample set into the model, and instruct the model to generate each annotation rule by learning the association relationship between each annotated text sample and its category label through a prompt text;

[0108] The forming unit 312 is used to form a rule base based on each annotation rule.

[0109] In one embodiment, the above sample categories include:

[0110] High confidence samples, medium confidence samples and low confidence samples.

[0111] In one embodiment, the first prediction result includes k probabilities corresponding to k predetermined categories; the determination unit 306 is specifically configured to:

[0112] When the maximum probability among the k probabilities falls within the first predetermined interval, determining that the first text sample belongs to a medium-confidence sample;

[0113] When the maximum probability is less than the lower limit value of the first predetermined interval, determining that the first text sample belongs to a low-confidence sample;

[0114] When the maximum probability is greater than the upper limit value of the first predetermined interval, it is determined that the first text sample belongs to a high-confidence sample.

[0115] In another embodiment, the first prediction result includes k probabilities corresponding to k predetermined categories; the determination unit 306 is specifically configured to:

[0116] Differences are calculated for k probabilities in pairs to obtain multiple differences;

[0117] When the minimum difference value among the multiple differences falls within the second predetermined interval, determining that the first text sample belongs to a medium-confidence sample;

[0118] When the minimum difference is less than the lower limit of the second predetermined interval, determining that the first text sample belongs to a low confidence sample;

[0119] When the minimum difference is greater than the upper limit of the second predetermined interval, it is determined that the first text sample is a high-confidence sample.

[0120] In yet another embodiment, the first prediction result includes k probabilities corresponding to k predetermined categories; the determination unit 306 is specifically configured to:

[0121] Based on k probabilities, calculate the first entropy value;

[0122] When the first entropy value falls within the third predetermined interval, determining that the first text sample belongs to a medium-confidence sample;

[0123] When the first entropy value is less than a lower limit value of the third predetermined interval, determining that the first text sample is a high-confidence sample;

[0124] When the first entropy value is greater than an upper limit value of the third predetermined interval, it is determined that the first text sample belongs to a low-confidence sample.

[0125] In one embodiment, the execution unit 308 includes:

[0126] A return submodule 3082 is used to return the first text sample to the sample pool to re-annotate it using the model if the first text sample belongs to a medium-confidence sample;

[0127] An adding submodule 3084 is used to manually add a category label to the first text sample when the first text sample is a low-confidence sample, and add the first text sample after the category label is added to the annotated sample set;

[0128] The determination submodule 3086 is used to, when the first text sample is a high-confidence sample, use the corresponding predicted label as its category label, and add the first text sample with the added category label to the annotated sample set.

[0129] In one embodiment, the determining unit 306 is further specifically configured to:

[0130] A target classifier is trained based on the first text sample and its predicted label, and each annotated text sample and its category label.

[0131] The functions of the functional units of the device in the above-mentioned embodiment of this specification can be implemented through the steps of the above-mentioned method embodiment. Therefore, the specific working process of the device provided by one embodiment of this specification will not be repeated here.

[0132] An apparatus for annotating text samples provided by an embodiment of the present specification can greatly improve the efficiency and accuracy of text sample annotation.

[0133] According to another embodiment, there is also provided a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 1 The method described.

[0134] According to another embodiment of the present invention, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the Figure 1 The method described.

[0135] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the medium or device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0136] The steps of the method or algorithm described in conjunction with the disclosure of this specification can be implemented in hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, register, hard disk, mobile hard disk, CD-ROM or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a server. Of course, the processor and the storage medium can also be present in the server as discrete components.

[0137] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0138] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26k20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0139] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, the present application does not exclude that with the development of computer technology in the future, the computer that implements the functions of the above embodiments may be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0140] Although one or more embodiments of the present specification provide method operation steps as described in the embodiments or flow charts, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps, and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements. For example, if the words first, second, etc. are used to represent the name, they do not represent any specific order.

[0141] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing one or more of the present specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0142] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0143] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0145] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0146] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0147] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, graphene storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0148] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0150] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0151] The above description is only an example of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. For those skilled in the art, one or more embodiments of the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims.

Claims

1. A method for labeling a text sample, comprising: Acquire a first text sample to be annotated from a sample pool; Using the model to label the first text sample to obtain a predicted label for the first text sample; Determining, by a target classifier, based on the first text sample and its predicted label, a sample category to which the first text sample belongs, which indicates a degree of confidence of the predicted label; Target processing corresponding to the sample category to which the first text sample belongs is performed on the first text sample.

2. The method according to claim 1, wherein: The determining the sample category to which the first text sample belongs includes: Based on the first text sample and its predicted label, training a target classifier; The sample category to which the first text sample belongs is determined according to a first prediction result output by the target classifier for the first text sample during or after the process of training the target classifier.

3. The method according to claim 1, wherein: The obtaining of the predicted label of the first text sample includes: Based on the first text sample, query the annotated sample set to obtain a second text sample matching the first text sample; each annotated text sample in the annotated sample set has a pre-annotated category label; The first text sample, the second text sample and their category labels are input into the model to obtain a predicted label of the first text sample.

4. The method according to claim 1, wherein: The obtaining of the predicted label of the first text sample includes: Based on the first text sample, a query rule library is obtained to obtain a target labeling rule that matches the first text sample; The first text sample and the target labeling rule are input into the model to obtain a predicted label of the first text sample.

5. The method according to claim 4, wherein: The rule base is constructed by the following steps: Inputting each annotated text sample and its category label in the annotated sample set into the model, and instructing the model to generate each annotating rule by learning the association relationship between each annotated text sample and its category label through a prompt text; The rule base is formed based on the various marking rules.

6. The method according to claim 2, wherein: The sample categories include: High confidence samples, medium confidence samples and low confidence samples.

7. The method according to claim 6, wherein: The first prediction result includes k probabilities corresponding to k predetermined categories; and determining the sample category to which the first text sample belongs includes: When the maximum probability among the k probabilities falls within a first predetermined interval, determining that the first text sample belongs to a medium-confidence sample; When the maximum probability is less than a lower limit value of the first predetermined interval, determining that the first text sample is a low-confidence sample; When the maximum probability is greater than an upper limit value of the first predetermined interval, it is determined that the first text sample is a high-confidence sample.

8. The method according to claim 6, wherein: The first prediction result includes k probabilities corresponding to k predetermined categories; and determining the sample category to which the first text sample belongs includes: Differences are calculated for the k probabilities in pairs to obtain a plurality of difference values; When the minimum difference value among the plurality of difference values ​​falls within a second predetermined interval, determining that the first text sample belongs to a medium-confidence sample; When the minimum difference is less than a lower limit of a second predetermined interval, determining that the first text sample is a low confidence sample; When the minimum difference is greater than an upper limit of the second predetermined interval, it is determined that the first text sample is a high-confidence sample.

9. The method according to claim 6, wherein: The first prediction result includes k probabilities corresponding to k predetermined categories; The determining the sample category to which the first text sample belongs includes: Based on the k probabilities, calculating a first entropy value; When the first entropy value falls within a third predetermined interval, determining that the first text sample belongs to a medium-confidence sample; When the first entropy value is less than a lower limit value of a third predetermined interval, determining that the first text sample is a high-confidence sample; When the first entropy value is greater than an upper limit value of a third predetermined interval, it is determined that the first text sample is a low-confidence sample.

10. The method according to claim 6, wherein: The performing target processing corresponding to the sample category to which the first text sample belongs includes: In the case where the first text sample is a medium-confidence sample, returning the first text sample to the sample pool to re-annotate it using the model; In the case where the first text sample is a low confidence sample, manually adding a category label to the first text sample, and adding the first text sample after the category label is added to the annotated sample set; In the case where the first text sample is a high-confidence sample, the corresponding predicted label is used as its category label, and the first text sample with the added category label is added to the labeled sample set.

11. The method according to claim 2, wherein: The training target classifier comprises: A target classifier is trained based on the first text sample and its predicted label, as well as each annotated text sample and its category label.

12. The method according to claim 2, wherein each method step is executed in a loop, and the loop termination condition includes any one of the following: The number of low-confidence samples in the sample pool is 0; The performance index of the target classifier reaches a first threshold; The improvement of the performance indicator of the target classifier is less than a second threshold.

13. A text sample annotation device, comprising: An acquisition unit, used for acquiring a first text sample to be annotated from a sample pool; A labeling unit, used to label the first text sample using the model to obtain a predicted label of the first text sample; a determination unit, configured to determine, by a target classifier, based on the first text sample and its predicted label, a sample category to which the first text sample belongs, which indicates a degree of confidence of the predicted label; An execution unit is used to perform target processing corresponding to the sample category to which the first text sample belongs.

14. A computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 12 is implemented.

Images