Text intelligent annotation method, system, intelligent terminal and storage medium

Abstract

The present invention discloses a method, system, intelligent terminal and storage medium for intelligent text annotation, wherein the method comprises: obtaining a text to be annotated; obtaining predicted annotation information of the text to be annotated based on a pre-trained text classification model, and annotating the text to be annotated based on the predicted annotation information, wherein the text classification model is pre-trained based on training sample data, and the training sample data includes real sample data and simulated sample data generated based on the real sample data. Compared with the prior art, the text classification model used in the present invention can be obtained by pre-training with real sample data and simulated sample data. Even when there is less real sample data or it is difficult to collect real sample data, a large amount of simulated sample data can be generated based on the small amount of real sample data collected, thereby improving the training effect of the text classification model, and further improving the effect and accuracy of text annotation.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a text intelligent annotation method, system, intelligent terminal and storage medium. Background Art

[0002] With the development of science and technology, text data in various fields continues to increase. Before processing or analyzing text data, each text needs to be annotated.

[0003] One existing approach is to train a model for text classification directly based on pre-collected and annotated sample data, and then implement text annotation based on the model. However, this existing approach suffers from the fact that the model training requires a large amount of sample data. This results in poor training results for domains or scenarios where sample data is insufficient (i.e., it's difficult to collect large amounts of text), hindering the effectiveness of text annotation.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0005] The main purpose of the present invention is to provide a text intelligent annotation method, system, intelligent terminal and storage medium, aiming to solve the problem in the prior art that when a training model is trained and text annotation is performed directly based on pre-real collected and annotated sample data, the training effect of the model is poor when there is insufficient sample data, which is not conducive to improving the text annotation effect.

[0006] In order to achieve the above-mentioned object, the first aspect of the present invention provides a text intelligent annotation method, wherein the above-mentioned text intelligent annotation method comprises:

[0007] Get the text to be annotated;

[0008] The predicted annotation information of the text to be annotated is obtained according to a pre-trained text classification model, and the text to be annotated is annotated according to the predicted annotation information, wherein the text classification model is pre-trained according to training sample data, and the training sample data includes real sample data and simulated sample data generated according to the real sample data.

[0009] Optionally, the above text classification model is pre-trained according to the following steps:

[0010] Acquire real sample data input by the target object, wherein the real sample data includes at least one real text;

[0011] Generating the simulated sample data based on the real sample data, displaying the unlabeled text to the target object, and labeling the corresponding unlabeled text according to the real labeling information input by the target object to convert the corresponding unlabeled text into labeled text, wherein the simulated sample data includes a plurality of simulated texts, the unlabeled text includes unlabeled real text and unlabeled simulated text, and the labeled text includes labeled real text and labeled simulated text;

[0012] The above-mentioned annotated text is input into the above-mentioned text classification model, and the predicted annotation information corresponding to the above-mentioned annotated text is generated according to the above-mentioned text classification model. According to the real annotation information and the predicted annotation information corresponding to the above-mentioned annotated text, the model parameters of the above-mentioned text classification model are adjusted, and the above-mentioned step of generating the above-mentioned simulated sample data according to the above-mentioned real sample data is continued until the preset training conditions are met to obtain a trained text classification model.

[0013] Optionally, generating the simulated sample data according to the real sample data includes:

[0014] For each real text in the above real sample data, multiple corresponding simulated texts are generated through direct generation steps based on multiple pre-trained text direct generation models;

[0015] For each annotated real text in the above real sample data, multiple corresponding simulated texts are generated through controlled generation steps based on multiple pre-trained text controlled generation models;

[0016] Among them, one of the above-mentioned text direct generation models or one of the above-mentioned text controlled generation models generates a corresponding simulated text based on an input real text.

[0017] Optionally, the direct generation step includes: inputting the real text into the text direct generation model, and using the text output by the text direct generation model as the simulated text corresponding to the real text;

[0018] The above-mentioned controlled generation steps include: inputting the above-mentioned annotated real text into the above-mentioned text classification model, obtaining the predicted annotation information corresponding to the above-mentioned annotated real text, inputting the above-mentioned annotated real text, the real annotation information corresponding to the above-mentioned annotated real text and the predicted annotation information into the above-mentioned text controlled generation model to generate controlled text, and using the text output by the above-mentioned text controlled generation model as the simulated text corresponding to the annotated real text.

[0019] Optionally, the displaying of the unlabeled text to the target object includes:

[0020] Obtaining target features for each of the unlabeled texts, wherein the target features include a category flag and a target latent vector. The category flag is used to distinguish the source of the unlabeled text. The target latent vector for the unlabeled real text is a preset default latent vector. The target latent vector for the unlabeled simulated text is a latent vector generated during the generation of the unlabeled simulated text by a direct text generation model or a controlled text generation model.

[0021] According to the above target features, a ranking score of each of the above unlabeled texts is obtained through a pre-set multi-layer fully connected network;

[0022] The unlabeled texts are displayed to the target object in sequence according to the ranking scores.

[0023] Optionally, the step of sequentially displaying the unlabeled texts to the target object according to the ranking scores includes: inputting the unlabeled texts into the text classification model, and obtaining candidate labels corresponding to the unlabeled texts according to predicted labeling information output by the text classification model;

[0024] Generating a plurality of display pages according to the ranking scores, wherein one of the display pages includes a plurality of the unlabeled texts and candidate tags corresponding to the respective unlabeled texts;

[0025] The above display pages are displayed to the above target objects in sequence.

[0026] Optionally, the above method further includes:

[0027] During the training process of the text classification model, after each update of the model parameters of the text classification model, all the annotated texts are input into the text classification model to obtain corresponding predicted annotation information, and the classification accuracy of the text classification model is calculated based on the actual annotation information and the predicted annotation information corresponding to the annotated texts;

[0028] Generate a classification effect display graph based on all classification accuracies obtained by the text classification model during training, display the classification effect display graph to the target object, and obtain training instructions fed back by the target object based on the classification effect display graph;

[0029] Among them, the above-mentioned preset training condition is that the above-mentioned training instruction is a training end instruction.

[0030] A second aspect of the present invention provides a text intelligent annotation system, wherein the text intelligent annotation system includes:

[0031] A module for obtaining text to be annotated, used to obtain text to be annotated;

[0032] The annotation module is used to obtain the predicted annotation information of the above-mentioned text to be annotated based on a pre-trained text classification model, and annotate the above-mentioned text to be annotated based on the above-mentioned predicted annotation information, wherein the above-mentioned text classification model is pre-trained based on training sample data, and the above-mentioned training sample data includes real sample data and simulated sample data generated based on the above-mentioned real sample data.

[0033] The third aspect of the present invention provides an intelligent terminal, which includes a memory, a processor, and a text intelligent annotation program stored in the memory and runnable on the processor. When the text intelligent annotation program is executed by the processor, it implements any step of the text intelligent annotation method.

[0034] A fourth aspect of the present invention provides a computer-readable storage medium, on which a text intelligent annotation program is stored. When the text intelligent annotation program is executed by a processor, the text intelligent annotation program implements any step of the above-mentioned text intelligent annotation method.

[0035] As can be seen from the above, in the scheme of the present invention, the text to be annotated is obtained; the predicted annotation information of the above text to be annotated is obtained according to the pre-trained text classification model, and the above text to be annotated is annotated according to the above predicted annotation information, wherein the above text classification model is pre-trained according to the training sample data, and the above training sample data includes real sample data and simulated sample data generated according to the above real sample data.

[0036] Compared to existing approaches that rely solely on pre-collected and annotated real sample data for model training and then perform text annotation based on the trained model, the text classification model used in this invention can be pre-trained using a combination of real sample data and simulated sample data. Even in areas where real sample data is scarce or difficult to collect, a large amount of simulated sample data can be generated based on the small amount of collected real sample data, thereby improving the training effect of the text classification model and, in turn, the effectiveness and accuracy of text annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 This is a flow chart of a method for intelligent text annotation provided by an embodiment of the present invention;

[0039] Figure 2This is a schematic diagram of a display page provided by an embodiment of the present invention;

[0040] Figure 3 This is a classification effect display diagram provided by an embodiment of the present invention;

[0041] Figure 4 This is a classification effect display diagram corresponding to a data set provided by an embodiment of the present invention;

[0042] Figure 5 This is a schematic diagram of a specific training process of a text classification model provided by an embodiment of the present invention;

[0043] Figure 6 This is a schematic diagram of the components of a text intelligent annotation system provided by an embodiment of the present invention;

[0044] Figure 7 This is a block diagram of the internal structure principle of a smart terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In the following description, specific details such as particular system structures and techniques are provided for purposes of illustration, not limitation, to facilitate a thorough understanding of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.

[0046] It will be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0047] It should also be understood that the terms used in the present specification are only for the purpose of describing particular embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0048] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0049] As used in this specification and the appended claims, the term "if" can be interpreted as meaning "when" or "upon" or "in response to determining" or "in response to being classified into," depending on the context. Similarly, the phrase "if it is determined" or "if it is classified into [described condition or event]" can be interpreted as meaning "upon determination" or "in response to determining" or "upon classification into [described condition or event]" or "in response to being classified into [described condition or event]," depending on the context.

[0050] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0051] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0052] With the development of science and technology, text data in various fields continues to increase. Before processing or analyzing text data, each text needs to be annotated.

[0053] One solution in the prior art is for users to manually annotate texts one by one, but this requires users to perform operations one by one, which is time-consuming and labor-intensive, and is not conducive to improving the efficiency of text annotation.

[0054] With the increasing maturity of artificial intelligence (AI) technology and theory, the application fields of AI algorithms are becoming wider and wider, so AI algorithms can also be used for text classification. For example, a pre-trained model can be used to perform text classification related tasks. Another solution in the prior art is to directly train a model for text classification based on real sample data that has been collected and annotated in advance, and implement text annotation based on the model. The problem with the prior art is that the model requires a large amount of sample data during the training process. For fields or scenarios where there is insufficient sample data (i.e., it is difficult to collect a large amount of text), the training effect of the model is not good, which is not conducive to improving the effect of text annotation.

[0055] At the same time, for each sample data collected, the user needs to manually label it one by one. In the existing technology, the data is usually displayed to the user for labeling in the order of collection, and there is a lack of distinction between the importance of sample data. In the limited labeling time, the user may label a large number of unimportant samples and fail to label important samples, which is not conducive to improving the efficiency and effect of model training.

[0056] Specifically, conventional text classification algorithms require a certain amount of labeled data relevant to the specific task. However, many text classification application scenarios are in the cold start phase, making it difficult to obtain sufficient data. Furthermore, while some business scenarios have a certain amount of data, much of it has not been effectively and efficiently manually labeled, making it impossible to directly use it to build text classification models. This means that in some business scenarios, the data required to train the model may exceed the actual available data, making it impossible to use the model for effective classification. Alternatively, a large amount of text data in these business scenarios has not been manually labeled and cannot be directly used for model training. Manually labeling each piece of data is time-consuming and inefficient.

[0057] In order to solve at least one of the above-mentioned problems, in the solution of the present invention, a text to be annotated is obtained; predicted annotation information of the text to be annotated is obtained according to a pre-trained text classification model, and the text to be annotated is annotated according to the predicted annotation information, wherein the text classification model is pre-trained according to training sample data, and the training sample data includes real sample data and simulated sample data generated according to the real sample data.

[0058] Compared to existing approaches that rely solely on pre-collected and annotated real sample data for model training and then perform text annotation based on the trained model, the text classification model used in this invention can be pre-trained using a combination of real sample data and simulated sample data. Even in areas where real sample data is scarce or difficult to collect, a large amount of simulated sample data can be generated based on the small amount of collected real sample data, thereby improving the training effect of the text classification model and, in turn, the effectiveness and accuracy of text annotation.

[0059] At the same time, when the text classification model is trained in the present invention, the unlabeled texts will be sorted according to their importance and then output to the user (i.e., the target object) for labeling, so that the user can first label the important (i.e., higher business value) texts in the sorting order, which is conducive to improving the training efficiency and effect of the text classification model, thereby achieving the purpose of more efficiently improving the classification effect.

[0060] Exemplary Methods

[0061] like Figure 1As shown, an embodiment of the present invention provides a method for intelligent text annotation. Specifically, the method includes the following steps:

[0062] Step S100: obtaining the text to be annotated.

[0063] The above-mentioned text to be annotated is text that needs to be annotated, and can be input by a user (such as a target object) or by other smart devices, which is not specifically limited here.

[0064] In the method of this embodiment, the text to be annotated is classified according to a pre-trained text classification model to obtain its corresponding category label (ie, predicted annotation information), and then the text to be annotated is marked with the corresponding category.

[0065] It should be noted that the training data used by the text classification model during the training process is data from the same field or the same scene as the text to be annotated, and the category label output by the text classification model depends on the actual annotation information input by the user during the training process. For example, if the user annotates the text in the corresponding training sample data into categories such as finance and entertainment during the training process, then the text to be annotated will also be annotated into categories such as finance and entertainment. In this embodiment, the above-mentioned text to be annotated is a text in the medical field, and if the user annotates the text in the corresponding training sample data into disease categories such as cerebral hemorrhage and hypertension during the training process, then when the classification model is used to annotate the above-mentioned text to be annotated, it will also be annotated into the corresponding disease category.

[0066] Step S200: Obtain predicted annotation information of the text to be annotated based on a pre-trained text classification model, and annotate the text to be annotated based on the predicted annotation information, wherein the text classification model is pre-trained based on training sample data, and the training sample data includes real sample data and simulated sample data generated based on the real sample data.

[0067] Specifically, during annotation, the text to be annotated is used as input data of a trained text classification model, and the corresponding output predicted annotation information is used as the category of the text to be annotated and annotated.

[0068] In this embodiment, the user-uploaded text data to be annotated is classified based on a pre-trained text classification model for a specific downstream text classification task. The text classification model can be a pre-trained BERT model. The pre-trained BERT model is trained for classification based on a training dataset and tested on a test dataset to obtain classification accuracy results. The model parameters of the pre-trained BERT model are iteratively updated, and the trained BERT model is ultimately used as the pre-trained text classification model CLS-Model. The training dataset and test dataset can be obtained by dividing the training sample data, and the specific division method is not limited.

[0069] It should be noted that compared with pre-training of models on general text corpora, the specific task of natural language text understanding or natural language text generation for a specific field dataset is a downstream task.

[0070] In this embodiment, the text classification model is pre-trained according to the following steps:

[0071] Acquire real sample data input by the target object, wherein the real sample data includes at least one real text;

[0072] Generating the simulated sample data based on the real sample data, displaying the unlabeled text to the target object, and labeling the corresponding unlabeled text according to the real labeling information input by the target object to convert the corresponding unlabeled text into labeled text, wherein the simulated sample data includes a plurality of simulated texts, the unlabeled text includes unlabeled real text and unlabeled simulated text, and the labeled text includes labeled real text and labeled simulated text;

[0073] The above-mentioned annotated text is input into the above-mentioned text classification model, and the predicted annotation information corresponding to the above-mentioned annotated text is generated according to the above-mentioned text classification model. According to the real annotation information and the predicted annotation information corresponding to the above-mentioned annotated text, the model parameters of the above-mentioned text classification model are adjusted, and the above-mentioned step of generating the above-mentioned simulated sample data according to the above-mentioned real sample data is continued until the preset training conditions are met to obtain a trained text classification model.

[0074] In this embodiment, the target object is the operator who controls the intelligent text annotation process. The real text is real sentence text pre-collected by the user, and in this embodiment, the real sample data input by the user includes at least one previously annotated real text to train the corresponding text classification model. In one application scenario, if all text in the real sample data input by the user is unannotated, the corresponding real text is first displayed to the user for annotation.

[0075] Specifically, the above-mentioned generation of the above-mentioned simulated sample data based on the above-mentioned real sample data includes:

[0076] For each real text in the above real sample data, multiple corresponding simulated texts are generated through direct generation steps based on multiple pre-trained text direct generation models;

[0077] For each annotated real text in the above real sample data, multiple corresponding simulated texts are generated through controlled generation steps based on multiple pre-trained text controlled generation models;

[0078] Among them, one of the above-mentioned text direct generation models or one of the above-mentioned text controlled generation models generates a corresponding simulated text based on an input real text.

[0079] It should be noted that the above-mentioned simulated sample data is generated by a pre-trained text generation model, which includes a text direct generation model and a text controlled generation model, which are respectively used to realize direct text generation and controlled text generation. In one application scenario, only the direct text generation method or the controlled text generation method can be used. In this embodiment, the two methods are combined, so that more text can be quickly generated by directly generating text, and the similarity between the generated text and the original real text can be improved by combining the controlled text generation method, thereby making the training effect of the text classification model better.

[0080] Furthermore, the above-mentioned text direct generation model and text controlled generation model can be obtained by training the same or different model structures in different ways. In this embodiment, the text generation model is obtained by pre-set offline process training. Specifically, based on a large amount of pre-set general text corpus, the basic text generation technology based on the pre-trained large model (such as ELMO, MASS, GPT, VAE) is used to fully train for this task to obtain multiple text generation model base-models (such as ELMO-base, MASS-base, GPT-base, VAE-base). The function of these base models is to receive a sentence of input text and then generate an approximate text based on this text. It should be noted that the input data corresponding to the text direct generation model and the text controlled generation model are different during training and use. The text direct generation model only needs to input the original real text, while the text controlled generation also needs to input other data such as the annotation data for the real text. In this embodiment, the real text is recorded as text-o and the generated simulated text is recorded as text-g.

[0081] Specifically, in this embodiment, the steps of text generation are repeatedly performed. When performing direct text generation, all unannotated text can be returned for text generation. When performing controlled text generation, data that has been annotated by the user can be returned for text generation.

[0082] Furthermore, the direct generation step includes: inputting the real text into the text direct generation model, and using the text output by the text direct generation model as the simulated text corresponding to the real text;

[0083] The above-mentioned controlled generation steps include: inputting the above-mentioned annotated real text into the above-mentioned text classification model, obtaining the predicted annotation information corresponding to the above-mentioned annotated real text, inputting the above-mentioned annotated real text, the real annotation information corresponding to the above-mentioned annotated real text and the predicted annotation information into the above-mentioned text controlled generation model to generate controlled text, and using the text output by the above-mentioned text controlled generation model as the simulated text corresponding to the annotated real text.

[0084] Specifically, when directly generating text, multiple base-models are used for text generation. Each text-o will generate multiple corresponding text-g sentences. In one application scenario, a real text text-o is: "I want to be a billionaire." The corresponding directly generated simulated text text-g is: "I will save every penny from now on." Another simulated text generated by another text generation model is: "I set many goals and work hard."

[0085] Specifically, when performing the step of controlled text generation, what needs to be used is annotated real text (which may include real text that has been annotated in advance, or real text that has been annotated by the user after being displayed to the user). Specifically, the text classification model in this embodiment (which may be the current text classification model in the training process) is used to predict the category of the real text to obtain predicted annotation information. Then, the corresponding part of the real annotation information that is inconsistent with the predicted annotation information and the corresponding real text are selected to form a data set F, and the data set F is used to perform text-controlled generation on the real text according to the controlled text generation technology to obtain the corresponding simulated text. In this embodiment, based on the controlled text generation, a simulated text that is closer to the real text can be obtained, thereby improving the training effect of the text classification model.

[0086] In one application scenario, the unlabeled text can be directly displayed to the target object so that the target object can label it. In this embodiment, all unlabeled texts are sorted according to their importance and then displayed to the user according to the sorting results, so that the user can label more important unlabeled texts first.

[0087] Specifically, presenting the unlabeled text to the target object includes: obtaining target features of each of the unlabeled texts, wherein the target features include a category flag and a target latent vector, the category flag is used to distinguish the source of the unlabeled text, the target latent vector of the unlabeled real text is a preset default latent vector, and the target latent vector of the unlabeled simulated text is a latent vector generated during the process of generating the unlabeled simulated text by a text direct generation model or a text controlled generation model;

[0088] According to the above target features, a ranking score of each of the above unlabeled texts is obtained through a pre-set multi-layer fully connected network;

[0089] The unlabeled texts are displayed to the target object in sequence according to the ranking scores.

[0090] Specifically, in this embodiment, the unlabeled texts used in the sorting include generated simulated texts and unlabeled real texts uploaded by users.

[0091] In this embodiment, all unlabeled texts are sorted, wherein the specific target features used in the sorting can be set and adjusted according to actual needs and are not specifically limited here.

[0092] In one application scenario, feature extraction is first performed on the unlabeled simulated text text-g. The extracted target features include: a category flag, which is used to distinguish whether the corresponding text is generated by a generative model or an input unlabeled real text; a target latent vector. If the unlabeled text is a simulated text generated by a text generation model, then during its generation process, the latent vector generated at the output layer of the text generation model base-model is used as the target latent vector; if the unlabeled text is an unlabeled real text provided by the user, a default latent vector is used as a placeholder, where the value of the default latent vector can be set and adjusted according to actual needs.

[0093] In another application scenario, other features can also be extracted during feature extraction. For example, text-g is input into the classification model cls-model to obtain the probability distribution vector V of the text in each category. The statistical characteristics of V (including variance, entropy, maximum value, and top-N value difference) are calculated; as well as the cross entropy between V and the category label of the corresponding text-o. The top-N value difference represents the difference between the largest or smallest N data points selected after sorting the probability distribution vector V.

[0094] Furthermore, we can also extract the vector of text-g corresponding to the CLS position in the CLS-model, where the vector of the CLS position is an intermediate vector output by the BERT type classification model, specifically representing the vector obtained by the weighted summation of all text word latent vectors by the self-attention medium in the BERT type classification model.

[0095] Furthermore, after extracting the target features, a corresponding ranking score (rank-score) is obtained through a multi-layer fully connected network (MLP-s) so that the ranking and display can be performed according to the ranking score. In this embodiment, each unlabeled text and its corresponding rank-score can also be combined into a pair as the ranking group result to facilitate sorting and improve processing efficiency.

[0096] The multi-layer, fully connected MLP-s network is pre-trained offline. Specifically, a training set for the MLP-s network can be constructed offline for a large number of relevant text classification tasks. The training set includes extracted target features and their corresponding annotations, specifically indicating whether they are beneficial for improving text classification results. Multiple features from the training set are combined and input into the multi-layer, fully connected MLP-s network, which is then fully trained to obtain the final appropriate network parameters.

[0097] In this embodiment, a display page is generated to provide better display and facilitate user annotation. Specifically, the above-mentioned unlabeled texts are displayed to the above-mentioned target object in sequence according to the above-mentioned ranking scores, including: inputting each of the above-mentioned unlabeled texts into the above-mentioned text classification model, and obtaining candidate labels corresponding to each of the above-mentioned unlabeled texts based on the predicted annotation information output by the above-mentioned text classification model;

[0098] Generating a plurality of display pages according to the ranking scores, wherein one of the display pages includes a plurality of the unlabeled texts and candidate tags corresponding to the respective unlabeled texts;

[0099] The above display pages are displayed to the above target objects in sequence.

[0100] Figure 2 This is a schematic diagram of a display page provided by an embodiment of the present invention. Figure 2As shown, in this embodiment, a display page may include unlabeled text, all labels corresponding to the unlabeled text, and candidate labels, wherein all the above labels are opened by a drop-down key, and the candidate labels are used to prompt the user of the most likely label category of the unlabeled text. At the same time, the above display page may also be provided with buttons such as the previous page, the next page, reset labeling, end labeling, effect display, and save task to facilitate user operation. Furthermore, the above display page may also display the current page and how many pages there are in total, and set a page jump button. It should be noted that in a display page, the unlabeled text is arranged from top to bottom in descending order of ranking scores, and the ranking score of the unlabeled text in one display page is greater than the ranking score of the unlabeled text in the next display page.

[0101] Specifically, the display page can be displayed via a webpage, app, computer terminal, etc. The candidate labels in the display page can be selected from the actual annotation information of the real text corresponding to the unannotated text, or can be selected from the category labels initially predicted by the classification model cls-model for the corresponding unannotated text (or unannotated simulated text), without specific limitation here.

[0102] It should be noted that, in the process of training the classification model, the preset training conditions may be that the loss calculated by the classification model on the test set is less than the preset loss threshold, the classification accuracy of the classification model is greater than the preset accuracy threshold, the number of iterations reaches the preset iteration threshold, or the user input to stop training is received.

[0103] In this embodiment, the method further includes: during the training process of the text classification model, after each update of the model parameters of the text classification model, inputting all the annotated texts into the text classification model and obtaining corresponding predicted annotation information, and calculating the classification accuracy of the text classification model based on the actual annotation information and the predicted annotation information corresponding to the annotated texts;

[0104] A classification effect display graph is generated based on all classification accuracies obtained by the above-mentioned text classification model during the training process, the above-mentioned classification effect display graph is displayed to the above-mentioned target object, and a training instruction based on the above-mentioned classification effect display graph is obtained; wherein the above-mentioned preset training condition is that the above-mentioned training instruction is a training end instruction.

[0105] Specifically, when training the text classification model, the pre-trained model BERT is classified and trained based on the annotated real text and the annotated simulated text, so that the model parameters are adjusted according to the output data to improve the classification accuracy of the model until the classification accuracy can meet the user's requirements. In this embodiment, after each model update, the accuracy of the updated text classification model is calculated. Specifically, the accuracy can be calculated based on the deviation between the predicted annotation information predicted by the model and the actual annotation information. For example, the number of text items with incorrect predictions divided by the total number of text items can be used as the accuracy.

[0106] It should be noted that as more and more text is annotated by users and more and more simulated texts are generated according to the text generation model, the number of annotated texts used to train the text classification model will also increase. However, since simulated texts may introduce errors, a larger number of annotated texts does not necessarily mean a better training effect (i.e., a higher accuracy). Therefore, in this embodiment, the calculated accuracy is stored and a classification effect display diagram is drawn based on the accuracy and the number of data items involved in the training.

[0107] Figure 3 This is a classification effect display diagram provided by an embodiment of the present invention, such as Figure 3 As shown, with the different numbers of texts involved in the training and / or the different number of model training times, the accuracy of the obtained text classification model is also different, and the accuracy is constantly changing, indicating that the classification effect (i.e., accuracy) of the text classification model is also constantly changing. The above classification effect display diagram is displayed to the user, and the user can enter the training instructions by clicking on the above classification effect display diagram, and decide whether to continue training according to the training instructions. If you choose to continue, return to the generation step of the simulated text and loop, otherwise the training is terminated, and after the training is completed, the user can select the appropriate training data batch and the text classification model parameters corresponding to the batch according to the classification effect display diagram and download them, so that the user can use the data and model corresponding to the node with the best effect, thereby obtaining a better text intelligent annotation effect.

[0108] It should be noted that the classification effect display diagrams obtained on different data sets are not exactly the same. In this embodiment, the above-mentioned text intelligent annotation method is tested on the Chinese standard classification task FewCLUE, and the changes in classification accuracy in different data sets with the increase of annotated data are obtained. In general, the accuracy shows an upward trend. Figure 4 This is a classification effect display diagram corresponding to a data set (specifically, the tnews data set) provided in an embodiment of the present invention. The classification effect display diagrams corresponding to other data sets are no longer displayed one by one.

[0109] In the embodiment of the present invention, the training process of the above text classification model is described in detail based on a specific application scenario. Figure 5 This is a schematic diagram of a specific training process of a text classification model provided by an embodiment of the present invention. Figure 5 As shown, a text generation model and a sorting model are pre-trained, wherein the text generation model and the sorting model are trained based on large-scale pre-training corpus. After the user uploads the real text, the text is directly generated or controlled based on the pre-trained text generation model, and then sorting, unlabeled text display, user annotation, text classification model parameter update, accuracy calculation, effect display and other steps are performed until the user chooses to end the training. It should be noted that the training process of the text generation model and the sorting model can be based on the offline training system. The processes such as unlabeled text display, user annotation and effect display can be implemented based on the front-end display system to enhance the user's interactive experience.

[0110] As can be seen from the above, the text classification model used in the intelligent text annotation method provided by the embodiments of the present invention can be pre-trained using a combination of real sample data and simulated sample data. Even in fields where real sample data is scarce or difficult to collect, a large amount of simulated sample data can be generated based on the small amount of collected real sample data, thereby improving the training effect of the text classification model and, in turn, the effectiveness and accuracy of text annotation.

[0111] Specifically, in this embodiment, a pre-trained text generation model can be used to generate text data, and then cutting-edge artificial intelligence-related technologies can be used to optimize the text, which is then provided to users for text annotation. The results of manual annotation can be effectively fed back to the text generation model to further optimize the generation efficiency and achieve the purpose of continuously optimizing the text classification results.

[0112] At the same time, when the text classification model is trained in the present invention, the unlabeled texts will be sorted according to their importance and then output to the user (i.e., the target object) for labeling, so that the user can first label the important (i.e., higher business value) texts in the sorting order, which is conducive to improving the training efficiency and effect of the text classification model, thereby achieving the purpose of more efficiently improving the classification effect.

[0113] Specifically, machine learning techniques can be used to prioritize unannotated text, improving the efficiency and effectiveness of user annotation, thereby enhancing text classification and annotation results. Furthermore, user annotations can provide controlled guidance on the direction of text generation, further improving classification results.

[0114] Exemplary devices

[0115] like Figure 6As shown in , corresponding to the above-mentioned text intelligent annotation method, an embodiment of the present invention further provides a text intelligent annotation system, and the above-mentioned text intelligent annotation system includes:

[0116] The to-be-annotated text acquisition module 310 is used to acquire the to-be-annotated text;

[0117] The annotation module 320 is used to obtain the predicted annotation information of the above-mentioned text to be annotated based on a pre-trained text classification model, and annotate the above-mentioned text to be annotated based on the above-mentioned predicted annotation information, wherein the above-mentioned text classification model is pre-trained based on training sample data, and the above-mentioned training sample data includes real sample data and simulated sample data generated based on the above-mentioned real sample data.

[0118] It should be noted that the specific structure and implementation of the above-mentioned text intelligent annotation system and its various modules or units can refer to the corresponding description in the above-mentioned method embodiment, and will not be repeated here.

[0119] It should be noted that the division method of the various modules of the above-mentioned text intelligent annotation system is not unique and is not used as a specific limitation here.

[0120] Based on the above embodiment, the present invention also provides an intelligent terminal, whose principle block diagram can be shown as follows: Figure 7 As shown. The above-mentioned intelligent terminal includes a processor, a memory, a network interface and a display screen connected through a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a text intelligent annotation program. The internal memory provides an environment for the operation of the operating system and the text intelligent annotation program in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with an external terminal through a network connection. When the text intelligent annotation program is executed by the processor, the steps of any one of the above-mentioned text intelligent annotation methods are implemented. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen.

[0121] Those skilled in the art will understand that Figure 7 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the solution of the present invention and does not constitute a limitation on the smart terminal to which the solution of the present invention is applied. The specific smart terminal may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0122] In one embodiment, a smart terminal is provided, which includes a memory, a processor, and a text intelligent annotation program stored in the memory and executable on the processor. When the text intelligent annotation program is executed by the processor, the steps of any one of the text intelligent annotation methods provided in the embodiments of the present invention are implemented.

[0123] An embodiment of the present invention further provides a computer-readable storage medium, on which a text intelligent annotation program is stored. When the text intelligent annotation program is executed by a processor, the steps of any one of the text intelligent annotation methods provided in the embodiments of the present invention are implemented.

[0124] It should be understood that the sequence numbers of the steps in the above embodiments do not imply a specific order of execution; the order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0125] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0126] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0127] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0128] In the embodiments provided herein, it should be understood that the disclosed systems / terminal devices and methods may be implemented in other ways. For example, the system / terminal device embodiments described above are merely illustrative. For example, the division of the modules or units described above is merely a logical functional division. In actual implementation, other division methods may be used, such as combining or integrating multiple units or components into another system, or omitting or not implementing certain features.

[0129] If the above-mentioned integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the above-mentioned computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0130] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A text intelligent annotation method, characterized in that: The text intelligent annotation method includes: Get the text to be annotated; Obtaining predicted annotation information of the text to be annotated based on a pre-trained text classification model, and annotating the text to be annotated based on the predicted annotation information, wherein the text classification model is pre-trained based on training sample data, the training sample data includes real sample data and simulated sample data generated based on the real sample data, and the real sample data includes pre-collected real text; The text classification model regenerates simulated sample data at each iteration during the pre-training process; During the pre-training process of the text classification model, after the simulated sample data is generated based on the real sample data, target features of each unlabeled text are obtained respectively, wherein the unlabeled text includes unlabeled real text and unlabeled simulated text, and the target features include target latent vectors. The target latent vector of the unlabeled real text is a preset default latent vector, and the target latent vector of the unlabeled simulated text is a latent vector generated in the process of generating the unlabeled simulated text by a text direct generation model or a text controlled generation model; according to the target features, a ranking score of each unlabeled text is obtained through a preset multi-layer fully connected network; and according to the ranking score, each unlabeled text is displayed to the target object in turn.

2. The text intelligent annotation method according to claim 1, characterized in that: The text classification model is pre-trained according to the following steps: Acquire real sample data input by the target object, wherein the real sample data includes at least one real text; Generating the simulated sample data based on the real sample data, displaying unlabeled text to the target object, and labeling the corresponding unlabeled text according to the real labeling information input by the target object to convert the corresponding unlabeled text into labeled text, wherein the simulated sample data includes a plurality of simulated texts, and the labeled texts include labeled real texts and labeled simulated texts; The annotated text is input into the text classification model, and predicted annotation information corresponding to the annotated text is generated according to the text classification model. The model parameters of the text classification model are adjusted according to the real annotation information and the predicted annotation information corresponding to the annotated text, and the step of generating the simulated sample data according to the real sample data is continued until the preset training conditions are met to obtain a trained text classification model.

3. The text intelligent annotation method according to claim 2, characterized in that: The generating of the simulated sample data according to the real sample data includes: For each real text in the real sample data, generating multiple corresponding simulated texts through a direct generation step based on multiple pre-trained text direct generation models; For each annotated real text in the real sample data, generating multiple corresponding simulated texts through a controlled generation step based on multiple pre-trained text controlled generation models; Among them, one of the text direct generation models or one of the text controlled generation models generates a corresponding simulated text based on an input real text.

4. The text intelligent annotation method according to claim 3, characterized in that: The direct generation step includes: inputting the real text into the text direct generation model, and using the text output by the text direct generation model as the simulated text corresponding to the real text; The controlled generation step includes: inputting the annotated real text into the text classification model, obtaining the predicted annotation information corresponding to the annotated real text, inputting the annotated real text, the real annotation information corresponding to the annotated real text, and the predicted annotation information into the text controlled generation model to generate controlled text, and using the text output by the text controlled generation model as the simulated text corresponding to the annotated real text.

5. The text intelligent annotation method according to claim 1, characterized in that: The target feature further includes a category mark, and the category mark is used to distinguish the source of the unlabeled text.

6. The text intelligent annotation method according to claim 1, characterized in that: The step of sequentially displaying the unlabeled texts to the target object according to the ranking scores includes: inputting the unlabeled texts into the text classification model, and obtaining candidate labels corresponding to the unlabeled texts according to the predicted labeling information output by the text classification model; generating a plurality of display pages according to the ranking scores, wherein one of the display pages includes a plurality of the unlabeled texts and candidate tags corresponding to each of the unlabeled texts; The display pages are displayed to the target object in sequence.

7. The text intelligent annotation method according to claim 2, characterized in that: The method further comprises: During the training process of the text classification model, after each update of the model parameters of the text classification model, all the annotated texts are input into the text classification model and corresponding predicted annotation information is obtained, and the classification accuracy of the text classification model is calculated based on the actual annotation information and the predicted annotation information corresponding to the annotated texts; Generating a classification effect display graph according to all classification accuracies obtained by the text classification model during the training process, displaying the classification effect display graph to the target object, and obtaining training instructions fed back by the target object based on the classification effect display graph; Wherein, the preset training condition is that the training instruction is a training end instruction.

8. A text intelligent annotation system, characterized in that: The text intelligent annotation system includes: A module for obtaining text to be annotated, used to obtain text to be annotated; a labeling module, configured to obtain predicted labeling information of the text to be labeled based on a pre-trained text classification model, and label the text to be labeled based on the predicted labeling information, wherein the text classification model is pre-trained based on training sample data, the training sample data including real sample data and simulated sample data generated based on the real sample data, and the real sample data including pre-collected real text; The text classification model regenerates simulated sample data at each iteration during the pre-training process; During the pre-training process of the text classification model, after the simulated sample data is generated based on the real sample data, the text intelligent annotation system is also used to: obtain target features of each unlabeled text respectively, wherein the unlabeled text includes unlabeled real text and unlabeled simulated text, the target features include target latent vectors, the target latent vectors of the unlabeled real texts are preset default latent vectors, and the target latent vectors of the unlabeled simulated texts are latent vectors generated in the process of generating the unlabeled simulated texts by a text direct generation model or a text controlled generation model; according to the target features, obtain the ranking scores of each unlabeled text through a preset multi-layer fully connected network; and display each unlabeled text to the target object in turn according to the ranking scores.

9. An intelligent terminal, characterized in that: The intelligent terminal includes a memory, a processor, and a text intelligent annotation program stored in the memory and executable on the processor. When the text intelligent annotation program is executed by the processor, the steps of the text intelligent annotation method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a text intelligent tagging program, and when the text intelligent tagging program is executed by the processor, the steps of the text intelligent tagging method according to any one of claims 1 to 7 are implemented.

Images