Text labeling method and device and storage medium

Abstract

The invention provides a text labeling method and device and a storage medium. According to the method, a large-scale language model is guided to be output through a created prompt template, and based on the prompt template, knowledge distillation is performed on a coarse labeling result output by the large-scale language model and a labeling result of a small-scale language model, so that corresponding distillation loss is obtained; and the prompt template of the large-scale language model and the small-scale language model are iteratively updated according to the distillation loss, so that the collaborative labeling result output by the updated large-scale language model and small-scale language model is more accurate and reliable, and the labeling efficiency is improved while the labor cost in the labeling process is reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a text annotation method, device and storage medium. Background Art

[0002] When processing text data, it is necessary to convert semi-structured or unstructured text data into structured data that can be processed by computers, among which the annotation of text data is the basis of structured processing.

[0003] Currently, existing annotation schemes for text data usually adopt deep learning methods, that is, using a single language model for automatic annotation.

[0004] However, using only a single language model for annotation will result in low reliability of the annotation results. Summary of the invention

[0005] The embodiments of the present application provide a text annotation method, device and storage medium to solve the defect that the existing solutions only use a single language model for automatic annotation when annotating text data, resulting in low reliability of the annotation results.

[0006] In a first aspect, an embodiment of the present application provides a text annotation method, comprising:

[0007] Get the target dataset;

[0008] Classifying each text data in the target data set by a clustering algorithm to obtain a precisely labeled data set and an unlabeled data set;

[0009] Acquire an expert knowledge set and a comparison data set according to the precise labeled data set, wherein the expert knowledge set is a randomly acquired labeled data set, and the comparison data set is a data set with the annotations deleted;

[0010] Creating an initial prompt template, and inputting the comparison data set into a large language model according to the data format indicated by the initial prompt template to output a coarsely annotated data set;

[0011] Acquire an alignment degree value according to the coarsely labeled data set and the expert knowledge set, and iteratively update the initial prompt template based on the alignment degree value to acquire a target prompt template;

[0012] Inputting the unlabeled data set into a small language model respectively, and inputting the unlabeled data set into a large language model according to the target prompt template, so as to obtain a first labeling result output by the small language model and a second labeling result output by the large language model;

[0013] A distillation loss is obtained according to the first annotation result and the second annotation result. If the distillation loss is less than a loss threshold, the first annotation result is determined as a text annotation result.

[0014] In a possible implementation, the data format indicated by the initial prompt template includes evaluation object and sentiment polarity;

[0015] The evaluation object is a recognition target in the text data, and the sentiment polarity is a positive evaluation or a negative evaluation of the recognition target extracted from the text data.

[0016] In a possible implementation, the obtaining the alignment value according to the coarsely labeled data set and the expert knowledge set includes:

[0017] Based on the expert knowledge set, the number of true positive examples, the number of false positive examples and the number of false negative examples are obtained in the coarsely annotated data set; wherein the number of true positive examples is the number of text data for which the large language model outputs a positive evaluation and the corresponding text annotations in the expert knowledge set are also positively evaluated; the number of false positive examples is the number of text data for which the large language model outputs a positive evaluation and the corresponding text annotations in the expert knowledge set are negatively evaluated; the number of false negative examples is the number of text data for which the large language model outputs a negative evaluation and the corresponding text annotations in the expert knowledge set are positively evaluated;

[0018] Obtaining a precision rate according to the number of true positive examples and the number of false positive examples, and obtaining a recall rate according to the number of true positive examples and the number of false negative examples;

[0019] The alignment degree value is obtained according to the precision and the recall.

[0020] In a possible implementation manner, obtaining the alignment degree value according to the precision and the recall rate includes:

[0021] Determine the product of the precision, the recall and a preset value;

[0022] Determine a sum of the precision and the recall;

[0023] The quotient of the product and the sum is determined as the alignment degree value.

[0024] In a possible implementation, the iteratively updating the initial prompt template based on the alignment degree value to obtain a target prompt template includes:

[0025] Detecting whether the alignment degree value is less than the alignment threshold;

[0026] If so, use the initial prompt template as the target prompt template;

[0027] If not, increase the number of evaluation objects and the number of sentiment polarities in the initial prompt template to update the initial prompt template;

[0028] A new alignment value is obtained based on the updated initial prompt template, and when the new alignment value is less than the alignment threshold, the updated prompt template is used as the target prompt template.

[0029] In a possible implementation manner, obtaining the distillation loss according to the first annotation result and the second annotation result includes:

[0030] Acquire an initial temperature scaling constant, and perform temperature scaling on the second annotation result based on the initial temperature scaling constant to obtain a soft annotation result;

[0031] Obtaining information divergence between the soft annotation result and the first annotation result, and determining it as a coarse annotation loss;

[0032] Obtaining a cross entropy loss between the first annotation result and an expert annotation result in the precise annotation data set, and determining the loss as the expert annotation result loss;

[0033] The distillation loss is obtained according to the rough labeling loss and the expert labeling result loss.

[0034] In a possible implementation manner, after obtaining the distillation loss according to the first annotation result and the second annotation result, the method further includes:

[0035] If the distillation loss is greater than or equal to the loss threshold, obtaining a new temperature scaling constant, wherein the new temperature scaling constant is greater than an initial temperature scaling constant;

[0036] Using the first annotation result as a new precisely labeled data set, and obtaining a new second annotation result based on the new precisely labeled data set;

[0037] Inputting the new second annotation result into the small language model to obtain a new first annotation result;

[0038] Based on the new temperature scaling constant, a new distillation loss is obtained according to the new first annotation result and the new second annotation result, and this cycle is repeated until the new distillation loss is less than the loss threshold, and the new first annotation result is determined as the text annotation result.

[0039] In a second aspect, an embodiment of the present application provides a text annotation device, including:

[0040] An acquisition module is used to acquire a target data set, classify each text data in the target data set by a clustering algorithm to obtain a precisely labeled data set and an unlabeled data set, and acquire an expert knowledge set and a comparison data set based on the precisely labeled data set, wherein the expert knowledge set is a randomly acquired labeled data set, and the comparison data set is a data set with the annotations deleted;

[0041] A processing module, configured to create an initial prompt template, input the comparison data set into a large language model according to the data format indicated by the initial prompt template to output a coarsely annotated data set, obtain an alignment value according to the coarsely annotated data set and the expert knowledge set, and iteratively update the initial prompt template based on the alignment value to obtain a target prompt template;

[0042] A control module is used to input the unlabeled data set into a small language model respectively, and input it into a large language model according to the target prompt template to obtain a first labeling result output by the small language model and a second labeling result output by the large language model, and obtain a distillation loss according to the first labeling result and the second labeling result. If the distillation loss is less than a loss threshold, the first labeling result is determined as a text labeling result.

[0043] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0044] The memory stores computer-executable instructions;

[0045] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0046] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0047] The text annotation method, device and storage medium provided in the present application guide the output of a large language model through a created prompt template, and based on the prompt template, perform knowledge distillation on the coarse annotation results output by the large language model and the annotation results of a small language model to obtain the corresponding distillation loss, and iteratively update the prompt template and the small language model according to the distillation loss, so that the annotation results output by the updated small language model are more accurate and reliable, and the labor cost in the annotation process is reduced while improving the annotation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0049] Figure 1 Schematic diagram of the text annotation method provided for this application Figure 1 ;

[0050] Figure 2 Schematic diagram of the text annotation method provided for this application Figure 2 ;

[0051] Figure 3 A schematic diagram of the structure of the text annotation device provided in this application;

[0052] Figure 4 A schematic diagram of the structure of the electronic device provided in this application.

[0053] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0054] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0055] With the rapid development of information technology, a large amount of text data has been accumulated in various industries. In order to effectively extract the rich information contained therein, most of the semi-structured or unstructured text data needs to be converted into structured data that can be processed by computers before a series of scientific research applications can be carried out. The annotation of text information is the basis for its structured processing. The familiar corpus obtained through text annotation is the basis for related research such as named entity recognition, relationship extraction, and sentiment analysis. Especially for text data in specific fields, such as finance, law, education, etc., high-quality annotation data has a decisive influence on model training and application effects.

[0056] However, traditional text annotation methods often use manual annotation or deep learning. First, manual annotation is time-consuming and labor-intensive, inefficient, and cannot meet the needs of large-scale text data annotation. In addition, the annotation quality depends entirely on the personal quality and carefulness of the annotator, and is prone to problems such as mislabeling and missing labels. With the continuous increase of text data and the continuous updating of domain knowledge, traditional manual annotation methods are difficult to adapt to rapidly changing needs. Secondly, deep learning methods usually use a single language model for automatic annotation, that is, only a large language model or a small language model is used, and the accuracy and reliability of the output results are low.

[0057] Therefore, the present application provides a text annotation method, device and storage medium, which guide the output of a large language model through a created prompt template, and based on the prompt template, perform knowledge distillation on the coarse annotation results output by the large language model and the annotation results of a small language model to obtain the corresponding distillation loss, and iteratively update the prompt template and the small language model according to the distillation loss, so that the annotation results output by the updated small language model are more accurate and reliable, and the labor cost in the annotation process is reduced while improving the annotation efficiency.

[0058] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0059] Figure 1 Schematic diagram of the text annotation method provided for this application Figure 1 ,like Figure 1 As shown, the method includes:

[0060] S101. Obtain a target data set, classify each text data in the target data set by a clustering algorithm to obtain a precisely labeled data set and an unlabeled data set, and obtain an expert knowledge set and a comparison data set based on the precisely labeled data set, wherein the expert knowledge set is a randomly acquired labeled data set, and the comparison data set is a data set with the labels deleted.

[0061] Specifically, the target dataset Where n is the size of the data set, y i ∈Y={1,2,…,C} is each text data x i Corresponding label categories, through unsupervised clustering algorithms such as Kmeans or K-Mediods algorithm, the text data in the target data set is divided into two data sets according to the preset field categories, and the data set with smaller data volume is annotated by experts to obtain a precisely labeled data set where m << n, such that the expert-precise-labeled data only accounts for a very small part of the target dataset. For the unlabeled dataset

[0062] S102. Create an initial prompt template, input the comparison dataset into a large language model according to the data format guided by the initial prompt template to output a rough labeled data group, obtain an alignment degree value based on the rough labeled data group and the expert knowledge set, and iteratively update the initial prompt template based on the alignment degree value to obtain a target prompt template.

[0063] Specifically, after obtaining the expert knowledge set and the comparison dataset, use PromptSync: Expert Knowledge Alignment (PEKA) to guide the large language model (LLM) to output results aligned with the expert knowledge. This way, there is no need to fine-tune the LLM parameters, which not only saves computing resources but also avoids the overfitting problem that may be caused by parameter updates. In addition, by optimizing the problem formulation rather than the model itself, PKEA enables the LLM to adapt to specific professional knowledge and task requirements more quickly, improving the accuracy and reliability of the LLM in a specific domain.

[0064] S103. Input the unlabeled dataset into a small language model respectively, and input it into the large language model according to the target prompt template to obtain a first labeled result output by the small language model and a second labeled result output by the large language model. Obtain a distillation loss based on the first labeled result and the second labeled result. If the distillation loss is less than the loss threshold, determine the first labeled result as the text labeling result.

[0065] Specifically, after obtaining the target prompt template of the large language model (LLM), based on the unlabeled dataset D unlabeled , obtain the corresponding labeled results through the large language model and the small language model (SLM) respectively. Perform temperature scaling on the second labeled result output by the large language model to obtain a soft labeled result. Calculate the distillation loss based on the soft labeled result, the first labeled result output by the small language model, and the expert-labeled result in the precise-labeled dataset. Based on the distillation loss, perform cyclic iteration on the small language model, and when the distillation loss is less than the loss threshold, use the labeled result output by the small language model as the text labeling result.

[0066] The text annotation method, device and storage medium provided in the embodiments of the present application guide the output of a large language model through a created prompt template, and based on the prompt template, perform knowledge distillation on the coarse annotation results output by the large language model and the annotation results of the small language model to obtain the corresponding distillation loss, and iteratively update the prompt template and the small language model according to the distillation loss, so that the annotation results output by the updated small language model are more accurate and reliable, and the labor cost in the annotation process is reduced while improving the annotation efficiency.

[0067] Figure 2 Schematic diagram of the text annotation method provided for this application Figure 2 ,like Figure 2 As shown, in this embodiment Figure 1 Based on the embodiment, a text annotation method is described in detail, and the method includes:

[0068] S201, obtaining a target data set, classifying each text data in the target data set by a clustering algorithm to obtain a precisely labeled data set and an unlabeled data set, and obtaining an expert knowledge set and a comparison data set based on the precisely labeled data set.

[0069] Specifically, after obtaining the precise labeled dataset, we determine the domain of expert knowledge and the specific tasks that need to be aligned, such as text classification, sentiment analysis, or entity recognition. We randomly collect annotated data sets from the precise labeled dataset and define the expert knowledge set E = (e1, e2, …, e m ), where each e i Represents an expert's annotation or knowledge output. A comparison dataset is obtained based on the expert knowledge set. The comparison dataset is a dataset with annotations deleted.

[0070] S202: Create an initial prompt template, and input the comparison data set into a large language model according to the data format indicated by the initial prompt template to output a coarsely annotated data set.

[0071] Specifically, after obtaining the expert knowledge set and the comparison data set, an initial prompt template P0 is created to guide the LLM output. The data format guided by the initial prompt template includes the evaluation object and the sentiment polarity. The evaluation object is the recognition target in the text data, and the sentiment polarity is the positive or negative evaluation of the recognition target extracted from the text data.

[0072] Furthermore, such as text annotation in the field of sentiment analysis, the initial prompt template P0 identifies the sentiment polarity and evaluation object in the text according to the expert knowledge set E. The expert knowledge set E includes text content and expert annotations. Text data such as x_i = "The camera of this mobile phone is really excellent, but the battery life is too short", y_i = [{"aspenct":"mobile phone camera","sentiment":"positive"},{"aspenct":"battery","sentiment":"negative"}], x_i and y_i form an e_i, which is a piece of expert knowledge.

[0073] S203. Based on the expert knowledge set, obtain the number of true positive examples, false positive examples, and false negative examples in the coarsely labeled data set, obtain the precision according to the number of true positive examples and the number of false positive examples, and obtain the recall according to the number of true positive examples and the number of false negative examples.

[0074] Specifically, after guiding LLM to generate the output coarsely labeled data set through the initial prompt template P0, it is then compared with expert knowledge to obtain the number of true positives, false positives, and false negatives;

[0075] The number of true positive examples is the number of text data with positive evaluations output by the large language model and the corresponding text annotations in the expert knowledge set are also positively evaluated; the number of false positive examples is the number of text data with positive evaluations output by the large language model and the corresponding text annotations in the expert knowledge set are negatively evaluated; the number of false negative examples is the number of text data with negative evaluations output by the large language model and the corresponding text annotations in the expert knowledge set are positively evaluated;

[0076] Furthermore, the precision (P, E) is obtained by the following formula:

[0077] precision(P,E)=TP / (TP+FP);

[0078] Among them, TP is the number of true positive examples, FP is the number of false positive examples, and the recall rate recall(P,E) is obtained by the following formula:

[0079] recall(P,E)=TP / (TP+FN);

[0080] Among them, TP is the number of true positive examples and FN is the number of false negative examples.

[0081] S204: Obtain the alignment degree value according to the precision and the recall.

[0082] Specifically, after obtaining the precision and recall, determine the product of the precision, the recall and a preset value, determine the sum of the precision and the recall, and determine the quotient of the product and the sum as the alignment degree value, which is obtained by the following formula:

[0083] A(P,E)=2×precision(P,E)×recall(P,E) / precision(P,E)+recall(P,E);

[0084] Among them, A(P,E) is the alignment value, which is used to measure the alignment degree between the output result of the large language model under the current prompt template and the expert knowledge set.

[0085] S205: Detect whether the alignment degree value is less than the alignment threshold, and if so, use the initial prompt template as the target prompt template.

[0086] Specifically, after obtaining the alignment value, if the detected alignment value is less than the alignment threshold, the initial prompt template is used as the target prompt template, that is, as the final Prompt-P * , so that the large language model can output the corresponding annotation results according to the guidance of the target prompt template.

[0087] S206: If not, increase the number of evaluation objects and the number of sentiment polarities in the initial prompt template to update the initial prompt template.

[0088] Specifically, if it is detected that the alignment value is greater than the alignment threshold, the prompt template is iteratively adjusted and updated according to the alignment value to reduce the alignment value, and a new alignment value is obtained based on the updated prompt template. This cycle is repeated until the new alignment value is less than the alignment threshold, or the performance improvement brought by the Prompt improvement is no longer significant, and the updated prompt template is used as the target prompt template.

[0089] S207: input the unlabeled data set into a small language model respectively, and input the unlabeled data set into a large language model according to the target prompt template, so as to obtain a first labeling result output by the small language model and a second labeling result output by the large language model.

[0090] Specifically, after obtaining the target prompt template that can guide the output of the large language model, create an SLM model with a simple structure and few parameters, such as RoBERTa (Robustly optimized BERT approach). For the unlabeled data set, the corresponding first labeling results are obtained through the small language model SLM and the large language model LLM respectively. And the second annotation result in and Indicates LLM and SLM for each data x i 's annotation.

[0091] S208: Obtain an initial temperature scaling constant, and perform temperature scaling on the second annotation result based on the initial temperature scaling constant to obtain a soft annotation result.

[0092] Specifically, the LLM-generated T(x i ) is a one-hot label, and it is necessary to "soften" the one-hot label by introducing a temperature scaling constant T to obtain the corresponding soft labeling result. Because in the temperature scaling process, the soft label generated by LLM usually contains more information than the one-hot label, which not only provides the information of the positive example, but also provides the probability of the negative example. For T(x i ) is as follows:

[0093]

[0094] Among them, when the temperature scaling constant T = 1, the soft label and the one-hot label are equivalent; when T>1, the relative probability difference of the soft label will decrease, which helps to reduce the difference between labels. T j ( x ) is the predicted probability of the LLM for the category.

[0095] S209: Obtain information divergence between the soft annotation result and the first annotation result, determine it as a coarse annotation loss, obtain a cross entropy loss between the first annotation result and an expert annotation result in the refined annotation data set, and determine it as an expert annotation result loss.

[0096] Specifically, after obtaining the soft annotation result T′(x i ), calculate and obtain the soft labeling result T′(x i ), and the first annotation result S(x i ) is the coarse labeling loss, which is calculated by i ) and S(x i ) is calculated by the KL divergence (Kullback-Leibler Divergence), that is, the information divergence. For the coarse labeling loss L soft , calculated by the following formula:

[0097]

[0098] Where C is the total number of categories, T′ j (x) and S j (x) are the predicted probabilities of the categories by LLM and SLM after temperature scaling;

[0099] Furthermore, for the expert annotation result loss L hard , calculated by the following formula:

[0100]

[0101] in, For SLM data x i The labeling result, y i is the expert annotation result, and C is the total number of categories.

[0102] S210: Obtain the distillation loss according to the rough labeling loss and the expert labeling result loss.

[0103] Specifically, after obtaining the coarse labeling loss and the expert labeling result loss, for the distillation loss L distill , calculated by the following formula:

[0104] L distill =λL s ' oft +(1-λ)L hard ;

[0105] Among them, λ is the preset hyperparameter, L soft is the coarse labeling loss, L distill Label the result loss for the experts.

[0106] S211: If the distillation loss is less than a loss threshold, determine the first annotation result as a text annotation result.

[0107] Specifically, after obtaining the distillation loss, check whether the distillation loss is less than the loss threshold. If so, the first annotation result of the small language model for the unlabeled data set is determined as the final collaborative annotation result, that is, the text annotation result. If not, the small language model needs to be iteratively updated.

[0108] S212: If the distillation loss is greater than or equal to the loss threshold, based on the new temperature scaling constant, obtain a new distillation loss according to the new first annotation result and the new second annotation result, and repeat this cycle until the new distillation loss is less than the loss threshold, and determine the new first annotation result as the text annotation result.

[0109] Specifically, if it is detected that the distillation loss is not less than the loss threshold, a new temperature scaling constant is obtained, the new temperature scaling constant is greater than the initial temperature scaling constant, the first annotation result is used as a new precise labeled data set, a new prompt template is obtained based on the new precise labeled data set, and the large language model output is guided by the new prompt template to obtain a new second annotation result;

[0110] Furthermore, the new second annotation result is input into the small language model to obtain a new first annotation result, so as to realize iterative update of the small language model. Based on the new temperature scaling constant, a new distillation loss is obtained according to the new first annotation result and the new second annotation result. This cycle is repeated until the new distillation loss is less than the loss threshold, or the performance improvement brought by the improvement is no longer significant. In this way, the new first annotation result is determined as the text annotation result, that is, the final collaborative annotation result.

[0111] The text annotation method, device and storage medium provided in the embodiments of the present application guide the output of a large language model through a created prompt template, and based on the prompt template, perform knowledge distillation on the coarse annotation results output by the large language model and the annotation results of the small language model to obtain the corresponding distillation loss, and iteratively update the prompt template and the small language model according to the distillation loss, so that the annotation results output by the updated small language model are more accurate and reliable, and the labor cost in the annotation process is reduced while improving the annotation efficiency.

[0112] Figure 3 A schematic diagram of the structure of the text annotation device provided in this application, such as Figure 3 As shown, the text annotation device 30 provided in this embodiment includes:

[0113] The acquisition module 301 is used to acquire a target data set, classify each text data in the target data set by a clustering algorithm to obtain a precisely labeled data set and an unlabeled data set, and acquire an expert knowledge set and a comparison data set according to the precisely labeled data set, wherein the expert knowledge set is a randomly acquired labeled data set, and the comparison data set is a data set with deleted labels;

[0114] The processing module 302 is used to create an initial prompt template, input the comparison data set into a large language model according to the data format indicated by the initial prompt template to output a rough annotation data set, obtain an alignment value according to the rough annotation data set and the expert knowledge set, and iteratively update the initial prompt template based on the alignment value to obtain a target prompt template;

[0115] The control module 303 is used to input the unlabeled data set into the small language model respectively, and input it into the large language model according to the target prompt template to obtain a first labeling result output by the small language model and a second labeling result output by the large language model, and obtain a distillation loss according to the first labeling result and the second labeling result. If the distillation loss is less than a loss threshold, the first labeling result is determined as a text labeling result.

[0116] Furthermore, in the acquisition module 301, the data format indicated by the initial prompt template includes an evaluation object and a sentiment polarity; the evaluation object is a recognition target in the text data, and the sentiment polarity is a positive evaluation or a negative evaluation of the recognition target extracted from the text data.

[0117] Further, the processing module 302 is specifically used to obtain the number of true positive examples, the number of false positive examples and the number of false negative examples in the coarsely annotated data set based on the expert knowledge set; wherein the number of true positive examples is the number of text data whose output is positively evaluated by the large language model and whose corresponding text annotations in the expert knowledge set are also positively evaluated; the number of false positive examples is the number of text data whose output is positively evaluated by the large language model and whose corresponding text annotations in the expert knowledge set are negatively evaluated; the number of false negative examples is the number of text data whose output is negatively evaluated by the large language model and whose corresponding text annotations in the expert knowledge set are positively evaluated;

[0118] Obtaining a precision rate according to the number of true positive examples and the number of false positive examples, and obtaining a recall rate according to the number of true positive examples and the number of false negative examples;

[0119] The alignment degree value is obtained according to the precision and the recall.

[0120] Further, the processing module 302 is specifically used to determine the product of the precision, the recall and a preset value;

[0121] Determine a sum of the precision and the recall;

[0122] The quotient of the product and the sum is determined as the alignment degree value.

[0123] Further, the processing module 302 is specifically used to detect whether the alignment degree value is less than the alignment threshold;

[0124] If so, use the initial prompt template as the target prompt template;

[0125] If not, increase the number of evaluation objects and the number of sentiment polarities in the initial prompt template to update the initial prompt template;

[0126] A new alignment value is obtained based on the updated initial prompt template, and when the new alignment value is less than the alignment threshold, the updated prompt template is used as the target prompt template.

[0127] Further, the processing module 302 is specifically configured to obtain an initial temperature scaling constant, and based on the initial temperature scaling constant, perform temperature scaling on the second annotation result to obtain a soft annotation result;

[0128] Obtaining information divergence between the soft annotation result and the first annotation result, and determining it as a coarse annotation loss;

[0129] Obtaining a cross entropy loss between the first annotation result and an expert annotation result in the precise annotation data set, and determining the loss as the expert annotation result loss;

[0130] The distillation loss is obtained according to the rough labeling loss and the expert labeling result loss.

[0131] Further, the control module 303 is further configured to obtain a new temperature scaling constant if the distillation loss is greater than or equal to the loss threshold, wherein the new temperature scaling constant is greater than the initial temperature scaling constant;

[0132] Using the first annotation result as a new precisely labeled data set, and obtaining a new second annotation result based on the new precisely labeled data set;

[0133] Inputting the new second annotation result into the small language model to obtain a new first annotation result;

[0134] Based on the new temperature scaling constant, a new distillation loss is obtained according to the new first annotation result and the new second annotation result, and this cycle is repeated until the new distillation loss is less than the loss threshold, and the new first annotation result is determined as the text annotation result.

[0135] The computer room energy-saving control device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0136] Figure 4 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 4 As shown, the electronic device 40 provided in this embodiment includes: at least one processor 401 and a memory 402. Optionally, the device 40 also includes a communication component 403. The processor 401, the memory 402 and the communication component 403 are connected via a bus 404.

[0137] In a specific implementation process, at least one processor 401 executes the computer-executable instructions stored in the memory 402, so that at least one processor 401 executes the above method.

[0138] The specific implementation process of the processor 401 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0139] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0140] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0141] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0142] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0143] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0144] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0145] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0146] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0147] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0148] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0149] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0150] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0151] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A text annotation method, characterized in that: include: Get the target dataset; Classifying each text data in the target data set by a clustering algorithm to obtain a precisely labeled data set and an unlabeled data set; Acquire an expert knowledge set and a comparison data set according to the precise labeled data set, wherein the expert knowledge set is a randomly acquired labeled data set, and the comparison data set is a data set with the annotations deleted; Creating an initial prompt template, and inputting the comparison data set into a large language model according to the data format indicated by the initial prompt template to output a coarsely annotated data set; Acquire an alignment degree value according to the coarsely labeled data set and the expert knowledge set, and iteratively update the initial prompt template based on the alignment degree value to acquire a target prompt template; Inputting the unlabeled data set into a small language model respectively, and inputting the unlabeled data set into a large language model according to the target prompt template, so as to obtain a first labeling result output by the small language model and a second labeling result output by the large language model; A distillation loss is obtained according to the first annotation result and the second annotation result. If the distillation loss is less than a loss threshold, the first annotation result is determined as a text annotation result.

2. The method according to claim 1, characterized in that The data format guided by the initial prompt template includes evaluation object and sentiment polarity; The evaluation object is a recognition target in the text data, and the sentiment polarity is a positive evaluation or a negative evaluation of the recognition target extracted from the text data.

3. The method according to claim 1, characterized in that: The obtaining of the alignment degree value according to the coarsely labeled data set and the expert knowledge set includes: Based on the expert knowledge set, the number of true positive examples, the number of false positive examples and the number of false negative examples are obtained in the coarsely annotated data set; wherein the number of true positive examples is the number of text data for which the large language model outputs a positive evaluation and the corresponding text annotations in the expert knowledge set are also positively evaluated; the number of false positive examples is the number of text data for which the large language model outputs a positive evaluation and the corresponding text annotations in the expert knowledge set are negatively evaluated; the number of false negative examples is the number of text data for which the large language model outputs a negative evaluation and the corresponding text annotations in the expert knowledge set are positively evaluated; Obtaining a precision rate according to the number of true positive examples and the number of false positive examples, and obtaining a recall rate according to the number of true positive examples and the number of false negative examples; The alignment degree value is obtained according to the precision and the recall.

4. The method according to claim 3, characterized in that The obtaining the alignment degree value according to the precision and the recall rate includes: Determine the product of the precision, the recall and a preset value; Determine a sum of the precision and the recall; The quotient of the product and the sum is determined as the alignment degree value.

5. The method according to claim 1, characterized in that The iterative updating of the initial prompt template based on the alignment degree value to obtain a target prompt template includes: Detecting whether the alignment degree value is less than the alignment threshold; If so, use the initial prompt template as the target prompt template; If not, increase the number of evaluation objects and the number of sentiment polarities in the initial prompt template to update the initial prompt template; A new alignment value is obtained based on the updated initial prompt template, and when the new alignment value is less than the alignment threshold, the updated prompt template is used as the target prompt template.

6. The method according to claim 1, characterized in that The obtaining the distillation loss according to the first marking result and the second marking result includes: Acquire an initial temperature scaling constant, and perform temperature scaling on the second annotation result based on the initial temperature scaling constant to obtain a soft annotation result; Obtaining information divergence between the soft annotation result and the first annotation result, and determining it as a coarse annotation loss; Obtaining a cross entropy loss between the first annotation result and an expert annotation result in the precise annotation data set, and determining the cross entropy loss as the expert annotation result loss; The distillation loss is obtained according to the rough labeling loss and the expert labeling result loss.

7. The method according to any one of claims 1 to 6, characterized in that: After obtaining the distillation loss according to the first marking result and the second marking result, the method further includes: If the distillation loss is greater than or equal to the loss threshold, obtaining a new temperature scaling constant, wherein the new temperature scaling constant is greater than an initial temperature scaling constant; Using the first annotation result as a new precisely labeled data set, and obtaining a new second annotation result based on the new precisely labeled data set; Inputting the new second annotation result into the small language model to obtain a new first annotation result; Based on the new temperature scaling constant, a new distillation loss is obtained according to the new first annotation result and the new second annotation result, and this cycle is repeated until the new distillation loss is less than the loss threshold, and the new first annotation result is determined as the text annotation result.

8. A text annotation device, characterized in that: include: An acquisition module is used to acquire a target data set, classify each text data in the target data set by a clustering algorithm to obtain a precisely labeled data set and an unlabeled data set, and acquire an expert knowledge set and a comparison data set based on the precisely labeled data set, wherein the expert knowledge set is a randomly acquired labeled data set, and the comparison data set is a data set with the annotations deleted; A processing module, configured to create an initial prompt template, input the comparison data set into a large language model according to the data format indicated by the initial prompt template to output a coarsely annotated data set, obtain an alignment value according to the coarsely annotated data set and the expert knowledge set, and iteratively update the initial prompt template based on the alignment value to obtain a target prompt template; A control module is used to input the unlabeled data set into a small language model respectively, and input it into a large language model according to the target prompt template to obtain a first labeling result output by the small language model and a second labeling result output by the large language model, and obtain a distillation loss according to the first labeling result and the second labeling result. If the distillation loss is less than a loss threshold, the first labeling result is determined as a text labeling result.

9. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images