A text label annotation method and device, electronic equipment and storage medium

Abstract

The embodiment of the application provides a text label annotation method and device, electronic equipment and storage medium, and relates to the technical field of data processing. The specific implementation scheme is: based on an annotation model, performing label annotation on each piece of text to be annotated to obtain candidate labels of each piece of text; based on the pieces of text and the candidate labels, constructing a plurality of sample pairs; analyzing the quality prediction results of each sample pair; based on the obtained quality prediction results, identifying each piece of target text in the pieces of text, which meets the to-be-optimized condition in terms of label annotation quality; performing label optimization on the current pieces of target text to obtain optimized candidate labels; after label optimization, determining the label annotation results of the pieces of text based on the current candidate labels of the pieces of text. It can be seen that the scheme of the application can effectively improve the accuracy of text label annotation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a text label annotation method and device, electronic equipment and a storage medium. BACKGROUND

[0002] In order to train a model for text classification, each piece of text required for model training needs to be labeled. The label annotation can refer to annotation of category labels. For example, for any text belonging to a product review, the text is labeled as one of the suggestion category, the good review category and the poor review category through label annotation.

[0003] In the prior art, a pre-trained annotation model is used to label each piece of text to obtain the label of each piece of text.

[0004] However, the annotation model needs to rely on a large amount of training data to ensure accuracy. However, in actual applications, the data used for training the annotation model often has the problem of uneven distribution or deficiency, resulting in low accuracy of the labels of each piece of text obtained by using the annotation model for label annotation. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a text label annotation method, device, electronic equipment and storage medium to effectively improve the accuracy of text label annotation. The specific technical solutions are as follows:

[0006] In a first aspect, the present application provides a text label annotation method, comprising:

[0007] Based on the annotation model, each piece of text to be annotated is labeled to obtain candidate labels of each piece of text;

[0008] Based on the candidate labels and the pieces of text, a plurality of sample pairs are constructed; each sample pair includes first text data and second text data, and each text data in the first text data and the second text data includes a text and a candidate label of the text;

[0009] Analyze the quality prediction result of each sample pair; the quality prediction result of each sample pair is used to represent the matching degree of the text and the candidate label in the first text data included in the sample pair, and the high or low degree of the matching degree of the text and the candidate label in the second text data included in the sample pair;

[0010] Based on the obtained quality prediction result, identify each piece of target text in the pieces of text whose label annotation quality meets the optimization condition;

[0011] optimizing the current target texts to obtain optimized candidate labels;

[0012] After the label optimization, the label annotation results of the texts are determined based on the current candidate labels of the texts.

[0013] In a second aspect, the present application provides a label annotation device for texts, comprising:

[0014] The annotation module is configured to perform label annotation on each text to be annotated based on the annotation model, and obtain candidate labels of the texts.

[0015] The construction module is configured to construct a plurality of sample pairs based on the texts and the candidate labels; each sample pair comprises first text data and second text data, and each text data in the first text data and the second text data comprises a text and a candidate label of the text.

[0016] The analysis module is configured to analyze the quality prediction result of each sample pair; the quality prediction result of each sample pair is used to represent the matching degree of the text and the candidate label in the first text data included in the sample pair, and the high or low degree of the matching degree of the text and the candidate label in the second text data included in the sample pair.

[0017] The identification module is configured to identify, based on the obtained quality prediction result, each target text in the texts whose label annotation quality meets the to-be-optimized condition.

[0018] The optimization module is configured to optimize the current target texts to obtain optimized candidate labels.

[0019] The determination module is configured to, after the label optimization, determine the label annotation results of the texts based on the current candidate labels of the texts.

[0020] In a third aspect, the present application provides an electronic device, comprising:

[0021] The memory is configured to store a computer program.

[0022] The processor is configured to execute the program stored in the memory, and implement the label annotation method for texts as described above.

[0023] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the label annotation method for texts as described above.

[0024] The embodiment of the present application further provides a computer program product comprising instructions which, when executed on a computer, cause the computer to perform the text label labeling method described above.

[0025] The embodiment of the present application has the following beneficial effects:

[0026] The scheme of the present application can use a labeling model to label labels of each piece of text to be labeled to obtain candidate labels of each piece of text, construct a plurality of sample pairs according to each piece of text and the candidate labels, analyze a relative relationship of a matching degree of the candidate labels of two texts in each sample pair to obtain a quality prediction result, use the quality prediction result of each sample pair to identify each piece of target text in the pieces of text, of which the label labeling quality meets a to-be-optimized condition, and perform optimization, and determine a label labeling result of each piece of text based on the current candidate labels of each piece of text after label optimization. It can be seen that the scheme of the present application can effectively identify each piece of target text in each piece of text, of which the label labeling quality meets the to-be-optimized condition, and perform label optimization, thereby effectively improving the accuracy of text label labeling.

[0027] Of course, implementing any product or method of the present application does not necessarily need to achieve all the advantages described above at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other embodiments can also be obtained by those skilled in the art based on these drawings.

[0029] FIG. 1(a) is a schematic diagram of a prior art text labeling method provided in an embodiment of the present application;

[0030] FIG. 1(b) is a schematic diagram of another prior art text labeling method provided in an embodiment of the present application;

[0031] Figure 2 FIG. 2 is a flowchart of a text label labeling method provided in an embodiment of the present application;

[0032] Figure 3 FIG. 3 is a schematic diagram of a target text meeting a to-be-optimized condition provided in an embodiment of the present application;

[0033] Figure 4 FIG. 4 is a schematic diagram of a second training data set provided in an embodiment of the present application;

[0034] Figure 5This is a schematic diagram illustrating the training process of a quality analysis model provided in an embodiment of this application;

[0035] Figure 6 This is a schematic diagram illustrating the principle of the labeling method provided in the embodiments of this application;

[0036] Figure 7 A schematic diagram of the structure of a text labeling device provided in an embodiment of this application;

[0037] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0038] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0039] Figures 1(a) and 1(b) show the existing methods for annotating text.

[0040] Specifically, as shown in Figure 1(a), in one method, the text is first manually labeled. As shown in Figure 1(a), the data of the text to be labeled includes: "It's so hard to use", "The sweeping function is pretty good", "The quality is good", "I hope it supports automatic", "The water tank is cracked". The labeling results can include: "Negative reviews", "Positive reviews", "Suggestions". The labeling results are then manually proofread and finally entered into the database.

[0041] Specifically, as shown in Figure 1(b), in one approach, the text is first tagged using an annotation model, then the annotation results are manually proofread, and finally stored in the database. The text data to be annotated as shown in Figure 1(b) includes: "It's so hard to use," "The sweeping function is great," "Good quality," "Hope it supports automatic operation," and "The water tank is cracked." The annotation results can include: "Negative reviews," "Positive reviews," and "Suggestions." The annotation model can be LLM (Large Language Model) or BERT (Bidirectional Encoder Representation from Transformers, a natural language processing model).

[0042] No matter in the way of FIG. 1(a) or in the way of FIG. 1(b), there will be the following defects: how much artificial participation, and how much time artificial proofreading needs to spend. Efficiency depends on the good or bad of the labeling effect, and if the labeling effect is not good, it will reduce the speed of proofreading. For the text that has been labeled and determined to be correct, it is not utilized in the subsequent labeling task. For difficult samples (truncation, incomplete semantics, ambiguity, etc.), each person's labeling is inconsistent.

[0043] In particular, the labeling model involved in FIG. 1(b) needs to rely on a large amount of training data to train to ensure accuracy. However, in actual application, the data utilized by the labeling model for training usually has the problem of uneven distribution or deficiency, thereby causing the confidence degree assigned to the label by the labeling model when labeling the label to be biased, and thereby the labels of each piece of text labeled exist the problem of low accuracy rate.

[0044] In the technical solution of the present application, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solution comply with relevant laws and regulations and do not violate public order and good customs.

[0045] The text label labeling method, device, electronic equipment and storage medium provided by the embodiments of the present application can effectively improve the accuracy of text label labeling.

[0046] First, a text label labeling method provided by the embodiments of the present application will be introduced. The text label labeling method provided by the embodiments of the present application can be applied to an electronic device. Illustratively, the electronic device can be a server, a terminal device, etc. The present application does not limit the specific form of the electronic device.

[0047] The text label labeling method provided by the embodiments of the present application can include:

[0048] Based on the labeling model, label each piece of text to be labeled to obtain the candidate label of each piece of text;

[0049] Based on the candidate label and the text, a plurality of sample pairs are constructed; each sample pair includes first text data and second text data, and each text data in the first text data and the second text data includes a text and a candidate label of the text;

[0050] Analyze the quality prediction result of each sample pair; the quality prediction result of each sample pair is used to represent the matching degree of the text and the candidate label in the first text data included in the sample pair, and the high or low degree of the matching degree of the text and the candidate label in the second text data included in the sample pair;

[0051] Based on the obtained quality prediction result, target text in each piece of text whose label annotation quality meets the to-be-optimized condition is identified;

[0052] The current target text is subjected to label optimization, and a candidate label after optimization is obtained.

[0053] After label optimization, the label annotation result of each piece of text is determined based on the current candidate label of each piece of text.

[0054] The scheme of the present application can use the annotation model to perform label annotation on each piece of text to be annotated to obtain a candidate label of each piece of text, and then construct a plurality of sample pairs according to each piece of text and the candidate label, analyze the relative relationship between the matching degrees of the candidate labels of the two texts in each sample pair to obtain a quality prediction result, identify the target text in each piece of text whose label annotation quality meets the to-be-optimized condition using the quality prediction result of each sample pair, and perform optimization. After label optimization, the label annotation result of each piece of text is determined based on the current candidate label of each piece of text. It can be seen that the scheme of the present application can effectively identify the target text in each piece of text whose label annotation quality meets the to-be-optimized condition, and perform label optimization, thereby effectively improving the accuracy of text label annotation.

[0055] As shown in Figure 2 a label annotation method for text provided by an embodiment of the present application includes:

[0056] S201, based on an annotation model, label annotation is performed on each piece of text to be annotated to obtain a candidate label of each piece of text.

[0057] The annotation model can be any annotation model capable of performing label annotation on text, and the present application does not limit the annotation model.

[0058] Specifically, each piece of text to be annotated is input to the annotation model together or separately, and a label generated by the annotation model for each text can be obtained. This label is the candidate label.

[0059] And the scheme of the present application can be used for texts in different fields, such as product evaluation texts, smart home interaction texts, video comment texts, etc. Any field with text label annotation can use the scheme of the present application for label annotation.

[0060] S202, based on each piece of text and the candidate label, a plurality of sample pairs are constructed; each sample pair includes first text data and second text data, and each text data in the first text data and the second text data includes a text and a candidate label of the text.

[0061] Each text and the candidate label corresponding to the text can constitute a text data. Any two text data can constitute a sample pair. In any sample pair, one text data is a first text data, and the other text data is a second text data.

[0062] Optionally, in an implementation, constructing a plurality of sample pairs based on the texts and the candidate labels comprises:

[0063] Each text data is combined with another text data to obtain a plurality of sample pairs. For example, if the number of texts is 100, there are 100 text data, and the combination of any two text data can obtain 4950 combinations.

[0064] Optionally, in an implementation, constructing a plurality of sample pairs based on the texts and the candidate labels comprises:

[0065] Each text data is divided into a plurality of groups.

[0066] Each text data in each group is combined with another text data to obtain a plurality of sample pairs.

[0067] S203, analyzing the quality prediction result of each sample pair. The quality prediction result of each sample pair is used to represent the matching degree of the text in the first text data and the candidate label in the sample pair, and the matching degree of the text in the second text data and the candidate label in the sample pair.

[0068] In an implementation, a large language model can be used to analyze the quality prediction result of each sample pair. Specifically, each sample pair and a prompt word can be input into the large language model, so that the large language model outputs the quality prediction result of each sample pair. A single sample pair can be input, or all sample pairs can be input together. The prompt word is used to instruct the large language model to perform quality prediction on the sample pair to obtain the quality prediction result.

[0069] The large language model is a deep learning model with a large number of parameters (usually tens of billions or even hundreds of billions) trained based on a large amount of data. The large language model can understand and generate human language, and can complete complex tasks such as reasoning and logical analysis. The scheme of the present application can use an existing large language model. The large language model can be used to understand and process text data.

[0070] In an implementation, a pre-trained target quality analysis model can be used to obtain the quality prediction result of each sample pair. The pre-trained target quality analysis model is described in detail in the embodiments described below.

[0071] S204, based on the obtained quality prediction results, identifying each target text in the texts whose label annotation quality meets the to-be-optimized condition.

[0072] Each target text in the texts whose label annotation quality meets the to-be-optimized condition is a text in the texts whose label annotation quality is low, i.e., the label is inaccurate, according to each quality prediction result.

[0073] S205, performing label optimization on the current target texts to obtain optimized candidate labels.

[0074] Each target text whose label annotation quality meets the to-be-optimized condition is a text whose label is inaccurate. For each target text whose label annotation quality meets the to-be-optimized condition, label optimization can be performed, i.e., a more accurate new label of the target text can be determined as an optimized candidate label.

[0075] S206, after label optimization, determining the label annotation results of the texts based on the current candidate labels of the texts.

[0076] For example, in an implementation manner, after label optimization, the current candidate labels of the texts can be directly used as the label annotation results of the texts. For example, in other implementation manners, one or more optimization processes can also be performed with the aid of the current candidate labels of the texts, which is also feasible, and the specific optimization process will be described in combination with other embodiments.

[0077] The scheme of the present application can use the annotation model to perform label annotation on the texts to be annotated to obtain candidate labels of the texts, construct a plurality of sample pairs according to the texts and the candidate labels, analyze the relative relationship between the matching degrees of the candidate labels of the two texts in each sample pair to obtain quality prediction results, identify each target text in the texts whose label annotation quality meets the to-be-optimized condition by using the quality prediction results of each sample pair, perform optimization, and determine the label annotation results of the texts based on the current candidate labels of the texts after label optimization. It can be seen that the scheme of the present application can effectively identify each target text in the texts whose label annotation quality meets the to-be-optimized condition, and perform label optimization, thereby effectively improving the accuracy of text label annotation.

[0078] Optionally, the determining of the label annotation results of the texts based on the current candidate labels of the texts comprises steps A1-A2.

[0079] Step A1, outputting a predetermined proportion of texts in the texts and corresponding candidate labels to indicate that manual inspection is performed on the predetermined proportion of texts.

[0080] The predetermined proportion can be set according to requirements, for example, 1%, which is not limited in the present application. The execution subject can randomly select the text and the corresponding candidate label of the predetermined proportion to provide to the detection personnel for manual inspection. In the randomly selected text and the corresponding candidate label, the corresponding candidate label is the current candidate label, that is, it can include the candidate label that has not been optimized, or it can include the candidate label that has been optimized.

[0081] Step A2, in response to obtaining the inspection result characterizing unqualification, returning to the step of analyzing the quality prediction result of each sample; otherwise, determining the current candidate label of each piece of text as the label annotation result of each piece of text.

[0082] The determination method of whether the inspection result is qualified can be set according to actual conditions. For example, it can be set that each candidate label in the output of each text and the corresponding candidate label is a correct label, which is qualified; or it can be set that the number of incorrect candidate labels in the output of each text and the corresponding candidate label is less than a specified number or less than a specified proportion, which is qualified.

[0083] The inspection result characterizing whether it is qualified can be directly uploaded by the detection personnel, or the detection personnel can upload the conclusion of whether each candidate label is correct, and then the execution subject determines the obtained conclusion according to whether each candidate label is correct.

[0084] If it is qualified, the current candidate label of each piece of text can be determined as the label annotation result of each piece of text, and if it is not qualified, the step of analyzing the quality prediction result of each sample is returned to optimize the label until the manual inspection is qualified.

[0085] It can be understood that if it is qualified, it means that the candidate label matching degree of the sampled text is high, which can further indicate that the candidate label matching degree of each text is high, and if it is not qualified, it means that the candidate label matching degree of the sampled text is not high, which further indicates that the candidate label matching degree of each text is not high.

[0086] And after returning to the step of analyzing the quality prediction result of each sample to optimize the label, the target text whose label annotation quality meets the optimization condition is determined again, which can be the text that does not belong to the text whose label annotation quality meets the optimization condition in the previous round.

[0087] In the manner of the present embodiment, whether the optimized text and the corresponding candidate label are qualified is judged by combining the manual sampling method, and if it is not qualified, it is continuously iterated, so that the candidate label is iteratively optimized, the candidate label of each text that passes the detection is taken as the label annotation result of each piece of text, and thus the accuracy of label annotation can be further improved.

[0088] Optionally, the step of performing label optimization on the current target texts to obtain the optimized candidate labels comprises steps B1-B2.

[0089] Step B1: For each current target text, at least one reference text satisfying a similarity condition is selected from texts other than the current target texts to obtain reference data of the target text, which comprises the selected reference text and a candidate label of the selected reference text.

[0090] It can be understood that each current target text is a target text whose label annotation quality meets the optimization condition, i.e., a target text with low label quality. The texts other than the current target texts are texts with high label quality.

[0091] The similarity condition is that the text similarity is greater than a specified value. That is, for each current target text, the text similarity between the target text and each text other than the current target texts can be calculated, and at least one reference text satisfying the similarity condition is selected based on the text similarity. At least one reference text satisfying the similarity condition can be selected in the manner of recognizing the text similarity by using a text similarity recognition model or in the manner of RAG (Retrieval-Augmented Generation).

[0092] Step B2: Based on the current target texts and the annotation prompt words of the reference data comprising each current target text, a target large language model is called to perform label annotation on each current target text based on the annotation prompt words to obtain the candidate labels of each optimized target text.

[0093] It can be understood that the reference data of each target text has high text similarity with the target text, and therefore the label content of the target text and the reference data of the target text are usually similar. Therefore, providing the reference data of each target text to the target large language model can enable the target large language model to analyze the label of the target text by taking the label of the reference data of the target text as a reference.

[0094] The annotation prompt words can comprise an instruction indicating that the large language model performs label annotation on the target text.

[0095] The target texts can be input to the large language model respectively or collectively. When input to the large language model collectively, the annotation prompt words can comprise the reference data of the target texts.

[0096] Therefore, the scheme of the embodiment enables the large language model to analyze the label of each target text not only based on the target text itself, but also based on the reference data of the target text, so that the analysis result is more accurate.

[0097] Of course, if there is no reference text satisfying the similar condition for any target text, the target large language model can only analyze and label the target text based on the target text itself.

[0098] Optionally, the target large language model is called based on the current target texts and the labeling prompt words containing the reference data of each current target text, so that the target large language model labels each current target text based on the labeling prompt words to obtain the candidate label of each optimized target text, including:

[0099] Step C1, the target large language model is called based on the current target texts and the labeling prompt words containing the reference data of each current target text, so that the target large language model labels each current target text based on the labeling prompt words and analyzes the labeling reason to obtain the candidate label of each optimized target text and the labeling reason.

[0100] Correspondingly, for each current target text, if the selected reference text has a labeling reason, the reference data of the target text containing the reference text also includes the labeling reason of the reference text.

[0101] For example, Figure 3 To meet the target texts of the to-be-optimized condition.

[0102] As Figure 3 shown, the data such as "It is very difficult to use", "The sweeping function is very good", "The quality is good", "I hope to support automatic", and "The water tank is cracked" are target texts meeting the to-be-optimized condition.

[0103] The target large language model analyzes the target texts meeting the to-be-optimized condition based on the reference texts and the reference reasons (i.e., the labeling reasons of the reference texts) to obtain the optimized labels (such as Figure 3 "negative evaluation", "positive evaluation", and "suggestion type") and the labeling reasons (such as Figure 3 XXX reasons) for labeling.

[0104] It can be understood that the reference text has a label annotation reason, which means that the reference text is a target text whose label annotation quality meets the to-be-optimized condition in the previous round, and after optimization, the reference text is a text whose label annotation quality does not meet the to-be-optimized condition. That is, through optimization, the reference text has a higher matching degree. Therefore, the label annotation reason of the reference text is to accurately determine the label annotation reason with a high matching degree, which can enable the large language model to analyze according to the reason, thereby obtaining a label with a high matching degree for the target text. Moreover, the label annotation reason analyzed in the historical round can provide support for label annotation in the next round, thereby achieving a self-evolution state.

[0105] Optionally, the analyzing the respective quality prediction results of each sample pair comprises:

[0106] Step D1, inputting each sample pair into a pre-trained target quality analysis model respectively to obtain the respective quality prediction result of each sample pair.

[0107] The target quality analysis model is trained based on each training data pair and the quality annotation label of each training data pair. Each training data pair comprises first training data and second training data, and each training data in the first training data and the second training data comprises a training text and a label of the training text.

[0108] For example, the target quality analysis model can be a Transformer (a deep learning model based on attention mechanism) model, and of course can also be other models, which are not limited in the present application.

[0109] The first text data and the second text data included in each sample pair can be vectorized. Specifically, for any text data, the text included in the text data and the candidate label of the text can be vectorized and spliced to obtain a vector.

[0110] Optionally, inputting each sample pair into a pre-trained target quality analysis model respectively to obtain the respective quality prediction result of each sample pair can comprise:

[0111] For each sample pair, the vector of the first text data corresponding to the sample pair and the vector of the second text data are input into the pre-trained target quality analysis model to obtain the quality prediction result of the sample pair.

[0112] The target quality analysis model of the present application evaluates the quality by comparing the matching degree of the label and the text in two text data, and the evaluation standard is relative. Compared with the label annotation by assigning confidence to the label, even if the model has bias due to the uneven or insufficient data distribution used for training, the model bias can be effectively reduced due to the comparison process of the two text data, in which one text data needs to be compared with another text data as the basis. Therefore, the model bias does not easily affect the comparison result, so that the quality evaluation result of the present application is more reliable.

[0113] The quality annotation label of each training data pair is used to represent the target relationship between the matching degree of the text and the label in the first training data included in the training data pair and the matching degree of the text and the label in the second training data included in the training data pair, which includes: higher, same or lower.

[0114] In an implementation manner, the quality annotation label is three of higher (representing that the label matching degree of the first text data is higher than the label matching degree of the second text data), same (representing that the label matching degree of the first text data is equal to the label matching degree of the second text data) or lower (representing that the label matching degree of the first text data is lower than the label matching degree of the second text data).

[0115] In an implementation manner, the quality annotation label is in the form of a score, and different scores represent the target relationship between the matching degree of the text and the label in the first training data included in the training data pair and the matching degree of the text and the label in the second training data included in the training data pair. For example, the quality prediction result is a score belonging to three intervals, which are: a first interval, a second interval and a third interval. The score of the first interval is a positive number, and the quality prediction result belonging to the first interval represents that the label matching degree of the first text data is higher than the label matching degree of the second text data, and the larger the score is, the higher the degree is. The score of the second interval is a number including 0 around both ends (i.e. including positive and negative numbers and 0), and the quality prediction result belonging to the interval represents that the label matching degree of the first text data is equal to the label matching degree of the second text data, and the larger the number greater than 0 is, the higher the degree that the label matching degree of the first text data is slightly higher than the label matching degree of the second text data is. The smaller the number less than 0 is, the higher the degree that the label matching degree of the second text data is slightly higher than the label matching degree of the first text data is. The score of the third interval is a negative number, and the quality prediction result belonging to the interval represents that the label matching degree of the second text data is higher than the label matching degree of the first text data, and the smaller the number is, the higher the degree is. The numbers included in the first interval, the second interval and the third interval are continuous numbers.

[0116] Optionally, the training manner of the target quality analysis model comprises:

[0117] perform first-stage training on the quality analysis model to be trained by using a first training data set to obtain a quality analysis model trained in the first stage;

[0118] perform second-stage training on the quality analysis model trained in the first stage by using a second training data set to obtain a target quality analysis model trained;

[0119] It can be understood that the amount of data of the public data set is greater than the amount of data of the specified field, and therefore, the quality analysis model capable of coping with various fields can be obtained by training using the public data set. The second-stage training can fine-tune the quality analysis model trained in the first stage by adjusting a small number of parameters to obtain a quality analysis model for the specified field.

[0120] It can be understood that the two-stage training method can obtain a quality analysis model for accurately analyzing sample pairs in the specified field.

[0121] Optionally, the training process in any one of the first-stage training and the second-stage training includes:

[0122] input each training data pair corresponding to the stage into the quality analysis model to be trained in the stage to obtain a quality prediction result of each training data pair; the quality prediction result of each training data pair is used to represent the matching degree of the text and the label in the first training data included in the training data pair, and the high-low situation compared with the matching degree of the text and the label in the second training data included in the training data pair;

[0123] determine the training data pair having the quality prediction result and the quality annotation label that do not satisfy the matching condition;

[0124] update the model parameters of the specified layer of the quality analysis model to be trained in the stage by using the backward propagation method based on the difference between the quality prediction result and the quality annotation label of the determined training data pair.

[0125] The quality prediction result and the quality annotation label of the training data pair satisfy the matching condition, which indicates that the prediction of the quality analysis model to be trained on the training data is accurate, and therefore, it is not necessary to adjust the parameters by using the training data.

[0126] If the quality prediction results and quality labels of the training data do not meet the matching conditions, it means that the quality analysis model to be trained is inaccurate in its prediction of the training data, and the parameters need to be adjusted using the training data.

[0127] For example, such as Figure 4 The diagram shown is a schematic representation of a second training dataset provided in this application. This second training dataset is for the evaluation of robotic vacuum cleaners.

[0128] like Figure 4 As shown, sample pairs 1, 2, and 3 are training samples from the second training dataset. In sample pair 1, the first text is "It's so hard to use," a negative review, and the second text is "It's terrible," a positive review, with a quality label of 1.0. In sample pair 2, the first text is "It's so hard to use," a negative review, and the second text is "The function is quite good," a positive review, with a quality label of 0.0. In sample pair 3, the first text is "It's not easy to use," a positive review, and the second text is "The function is quite good," a positive review, with a quality label of -1.0.

[0129] For example, such as Figure 5 The diagram shown is a schematic representation of the training process of the quality analysis model in this application.

[0130] like Figure 5 As shown, step 1 is the first stage of training, and step 2 is the second stage of training.

[0131] The open-source data is the first training dataset, and the vertical data is the second training dataset. Training data pairs are data pairs constructed using data from either the first or second training dataset. The encoder is part of the quality analysis model, and the loss function corresponds to backpropagation parameter adjustment. The transfer weights refer to the quality analysis model trained in the first stage, which serves as the model to be trained in the second stage. The process of backpropagation parameter adjustment is described in detail as indicated by the dotted line. Based on the difference between the model output and the original labels, training data pairs that meet expectations are assigned gradients to zero (i.e., no backpropagation is performed), while training data pairs that do not meet expectations can be used for gradient backpropagation to adjust the model parameters.

[0132] The method in this embodiment optimizes the parameter update process of model training. It uses only the existing quality prediction results and training data pairs whose quality labels do not meet the matching conditions for backpropagation to update the model parameters, thereby effectively improving the model training efficiency and enhancing the robustness of the model.

[0133] Optionally, in one implementation, the step of identifying each target text in each text whose labeled quality meets the optimization conditions based on the obtained quality prediction results includes:

[0134] Step E1: Based on the quality prediction results of each sample, sort the texts according to the matching degree from high to low to obtain the first sorting result;

[0135] Step E2: From the first sorting results, select a predetermined proportion of texts that are ranked lower to obtain each target text whose labeling quality meets the optimization conditions.

[0136] The method in this embodiment corresponds to the above method of combining each text data in pairs to obtain multiple sample pairs, that is, the sample pairs are obtained by combining each text in pairs.

[0137] The predetermined ratio can be set according to the actual situation.

[0138] In this implementation, sample pairs are obtained by combining each text data in pairs. Based on each quality prediction result, the relationship between the degree of label matching between any two texts can be determined. Thus, each text is effectively sorted according to the degree of label matching. Since the sorting is based on the relative relationship between each text, the sorting result is more accurate.

[0139] Optionally, in one implementation, the step of identifying each target text in each text whose labeled quality meets the optimization conditions based on the obtained quality prediction results includes:

[0140] Step F1: For each text group obtained after grouping the texts, based on the quality prediction results of each sample, sort the texts in the text group according to the matching degree from high to low to obtain a second sorting result. From the second sorting result, select a predetermined proportion of texts that are sorted lower to obtain the target texts corresponding to the text group whose labeling quality meets the optimization conditions.

[0141] This embodiment does not limit the construction of sample pairs. When the sample pairs are constructed by combining each text data in pairs, the method before step F1 further includes: grouping the texts to obtain text groups.

[0142] Therefore, for each text group, the texts in that group are ranked based on the quality prediction results of the sample pairs that can be formed by each text in that group. In other words, although all sample pairs that can be formed by all texts have quality prediction results, the ranking of each text in each group can be completed using only the quality prediction results of the sample pairs that can be formed by each text in that group.

[0143] When the sample pair is constructed by dividing each text data into multiple groups, and combining each text data in each group to obtain multiple sample pairs, each text group obtained after grouping the text in step F1 is the group divided during sample pair construction, and each sample pair in each group corresponds to a quality prediction result.

[0144] The predetermined ratio can be set according to the actual situation.

[0145] In this implementation, the text data can be divided into multiple groups. Within each group, the text data is sorted based on the quality prediction results of the sample pairs, resulting in a ranking of the label matching degree for each piece of text data within that group. This method allows analysis only of text data within a group, eliminating the need to analyze two pieces of text data belonging to different groups, thus effectively improving sorting efficiency.

[0146] Regardless of whether steps E1-E2 or step F1 are implemented, the principle of sorting texts based on quality prediction results is the same: the degree of matching between the tags of each pair of texts (all texts or texts belonging to a group) is determined, and thus the sorting results of the tag matching degree of each text data can be obtained.

[0147] The following section details the process of ranking texts based on quality prediction results.

[0148] For example, in one implementation, the quality prediction result is divided into three results: greater than, less than, and equal to.

[0149] The following describes the sorting of four texts in this implementation method. Table 1 below shows the labels, text content, candidate tags, and real tags of the four texts. Each text in Table 1 is specific to the mobile phone domain.

[0150] Table 1

[0151]

[0152] For ease of explanation, we use the labels 1-4 to represent each piece of text data (i.e., the text and its candidate labels).

[0153] The quality prediction results for both text 1 and text 2 are: text 1 equals text 2; the quality prediction results for both text 1 and text 3 are: text 1 is greater than text 3; the quality prediction results for both text 1 and text 4 are: text 1 is greater than text 4; the quality prediction results for both text 2 and text 3 are: text 2 is greater than text 3; the quality prediction results for both text 2 and text 4 are: text 2 is greater than text 4; the quality prediction results for both text 3 and text 4 are: text 3 is greater than text 4.

[0154] The sorting result is: Text1 = Text2 > Text3 > Text4.

[0155] For example, in one implementation, the quality prediction result is a score belonging to three intervals: a first interval, a second interval, and a third interval. The score in the first interval is positive, indicating that the quality prediction result belonging to the first interval signifies a higher degree of label matching for the first text data than for the second text data, with a higher score indicating a greater degree of higher matching. The score in the second interval is a series of consecutive numbers (including both positive and negative numbers) around 0, indicating that the quality prediction result belonging to this interval signifies a comparable degree of label matching for the first and second text data, with a higher number indicating a slightly higher degree of label matching for the first text data than for the second, and a lower number indicating a slightly higher degree of label matching for the second text data than for the first. The score in the third interval is negative, indicating that the quality prediction result belonging to this interval signifies a higher degree of label matching for the second text data than for the first text data, with a lower number indicating a greater degree of higher matching. The numbers in the first, second, and third intervals are consecutive.

[0156] For example, the first interval can be [0.5, 1.0), the second interval can be (-0.5, 0.5), and the third interval can be (-1.0, -0.5).

[0157] It is understandable that there can be various types of tags, and a text can actually correspond to multiple tags, meaning that multiple tags can correctly match the text. However, the degree of matching for each tag may vary. For example, in a smart home scenario, the text of the dialogue is: "Too bright, dim it a bit." Both the tags "device control" and "brightness control" are correct tags, but the tag accuracy of "brightness control" is higher than that of "device control," so the tag matching degree of "brightness control" is higher than that of "device control."

[0158] For example, if the quality prediction result of the sample pair consisting of text 1 and text 2 is 0.8, and the quality prediction result of the sample pair consisting of text 1 and text 3 is 0.7, it means that the label matching degree of text 1 is higher than that of text 2, and the label matching degree of text 1 is higher than that of text 3. Since 0.8 is greater than 0.7, it means that the label matching degree of text 2 is greater than that of text 3. In this case, the label matching degree relationship between text 2 and text 3 can be obtained without relying on the quality prediction result of the sample pair consisting of text 2 and text 3.

[0159] The quality prediction results in this embodiment are presented as scores for each interval, which can effectively characterize the relationship between the matching degree of labels between texts, improve the distinguishability of the label matching degree of each text (especially the distinguishability when the label matching degree of two text data is similar), and make the ranking more accurate.

[0160] The following describes the sorting of four texts in this implementation method. Table 2 below shows the labels, text content, candidate tags, and real tags of the four texts. The texts in Table 2 are specific to the smart home field.

[0161] Table 2

[0162]

[0163] For ease of explanation, the letters AD are used to represent each piece of text data (i.e., the text and its candidate labels).

[0164] For A and B, the quality prediction result is 0.3 (indicating that their qualities are similar, and A is slightly higher than B); for A and C, the quality prediction result is 1.0 (indicating that A is better than C); for A and D, the quality prediction result is 0.0 (indicating that A and D are of the same quality); for B and C, the quality prediction result is 0.7 (indicating that B is better than C); for B and D, the quality prediction result is -0.3 (indicating that their qualities are similar, and D is slightly better than B); and for C and D, the quality prediction result is -1.0 (indicating that D is better than C).

[0165] Based on the above prediction results, the order of the text data can be determined as: A=D>B>C.

[0166] As can be seen, the solution in this embodiment can effectively sort each piece of text data according to the degree of matching of the text tags.

[0167] Below, through a specific embodiment, combined with Figure 6The schematic diagram shown illustrates the principle of the text tagging method, providing a detailed introduction to the text tagging method. Specifically, the text tagging method provided in this application embodiment may include the following process:

[0168] Step 1: Use the pre-labeling model to pre-label all current data to obtain basic labeled data.

[0169] like Figure 6 As shown, the data "It's so hard to use," "The sweeping function is pretty good," "The quality is good," "I hope it supports automatic operation," and "The water tank is cracked" are evaluation texts of the robot vacuum cleaner, and these are also the texts to be labeled in this embodiment. The pre-labeling model can be LLM or BERT. Figure 6 The annotation results are the candidate tags for each text, such as: negative evaluation, positive evaluation, and suggestion.

[0170] Step 1 corresponds to S201 above.

[0171] Step 2: Sort the labeled data. The top 70% of the data is considered high-quality data, and the bottom 30% is considered low-quality data.

[0172] like Figure 6 As shown, the top 70% of the data is considered high-quality data, and the labels can be considered acceptable. The bottom 30% of the data is considered low-quality data, and the labels can be considered unacceptable.

[0173] Step 2 corresponds to S202-S204 above. The bottom 30% of the low-quality data are the target texts in each of the above texts whose labeled quality meets the optimization conditions.

[0174] Step 3: Relabel the 30% of low-quality data and retrieve the high-quality data.

[0175] Step 3 corresponds to S205 above. High-quality data is retrieved for use as reference data.

[0176] You can use the RAG method for searching. Figure 6 The search in the middle corresponds to step 3.

[0177] Step 4: Have LLM-Thinking create reference annotations and provide reasons for the annotations. At this point, the archive will include both the annotations and their reasons.

[0178] LLM-Thinking is essentially a large language model. Step 4 is... Figure 6 The LLM-reference annotation and LLM-reasoning explanation are included.

[0179] Step 4 corresponds to step C1 above, and will not be elaborated on here.

[0180] Step 5: Manually sample 1% to determine if it is qualified. If it is not qualified, continue the iteration. If it is qualified, it means that the text labeling is completed.

[0181] Step 5 corresponds to steps A1-A2 above.

[0182] The solution in this embodiment can effectively improve the accuracy of text labeling.

[0183] Based on the above-described embodiments of the text labeling method, this application also provides a text labeling device. Figure 7 This is a schematic diagram of the structure of a text labeling device provided in an embodiment of this application, as shown below. Figure 7 As shown, the text labeling device may include:

[0184] The annotation module 701 is used to label each text to be annotated based on the annotation model, and obtain candidate labels for each text.

[0185] The construction module 702 is used to construct multiple sample pairs based on the texts and candidate tags; each sample pair includes first text data and second text data, and each text data in the first text data and second text data includes a text and a candidate tag for that text;

[0186] Analysis module 703 is used to analyze the quality prediction results of each sample pair; the quality prediction results of each sample pair are used to characterize: the degree of matching between the text and candidate labels in the first text data included in the sample pair, compared with the degree of matching between the text and candidate labels in the second text data included in the sample pair.

[0187] The identification module 704 is used to identify, based on the obtained quality prediction results, each target text in the text whose label marks meet the optimization conditions.

[0188] Optimization module 705 is used to optimize the labels of each current target text to obtain optimized candidate labels;

[0189] The determination module 706 is used to determine the labeling result of each text based on the current candidate labels of each text after label optimization.

[0190] The proposed solution utilizes a labeling model to annotate each text to be labeled, obtaining candidate labels for each text. Then, based on each text and its candidate labels, multiple sample pairs are constructed. The relative matching degree of the candidate labels between the two texts in each sample pair is analyzed to obtain a quality prediction result. Using the quality prediction results of each sample pair, target texts whose labeling quality meets the optimization criteria are identified and optimized. After label optimization, the labeling result of each text is determined based on its current candidate labels. Therefore, the proposed solution can effectively identify target texts whose labeling quality meets the optimization criteria and optimize their labels, thereby significantly improving the accuracy of text labeling.

[0191] Optionally, the determining module includes:

[0192] The output unit is used to output a predetermined proportion of the text in each text and the corresponding candidate tags, so as to indicate that the predetermined proportion of text is subject to manual inspection.

[0193] A determination unit is used to, in response to obtaining a test result indicating non-compliance, return to the step of analyzing the quality prediction result for each sample; otherwise, determine the current candidate label for each text as the labeling result for each text.

[0194] Optionally, the optimization module includes:

[0195] The selection unit is used to select at least one reference text that meets similar conditions from the texts other than the current target texts for each current target text, so as to obtain reference data of the target text, which includes the selected reference text and the candidate tags of the selected reference text.

[0196] The calling unit is used to call the target large language model based on each current target text and the annotation prompt words containing reference data of each current target text, so that the target large language model can label each current target text based on the annotation prompt words to obtain optimized candidate labels for each target text.

[0197] Optionally, the analysis module includes:

[0198] The input unit is used to input each sample pair into the pre-trained target quality analysis model to obtain the quality prediction result for each sample pair.

[0199] The target quality analysis model is trained based on each training data pair and the quality label of each training data pair; each training data pair includes first training data and second training data, and each training data in the first training data and the second training data includes a training text and the label of the training text.

[0200] The quality label of each training data pair is used to characterize: the degree of label matching between the text and labels in the first training data included in the training data pair, and the target relationship between the degree of matching between the text and labels in the second training data included in the training data pair, wherein the target relationship includes: higher than, the same as, or lower than.

[0201] Optionally, the training method for the target quality analysis model includes:

[0202] Using the first training dataset, the quality analysis model to be trained is trained in the first stage to obtain the quality analysis model after the first stage of training; wherein, the first training dataset includes multiple training data pairs constructed based on a predetermined public dataset;

[0203] Using the second training dataset, the quality analysis model trained in the first stage is trained in the second stage to obtain the target quality analysis model after training; wherein, the second training dataset includes multiple training data pairs constructed based on texts in a specified domain, and the specified domain is the domain to which each text belongs.

[0204] Optionally, the training process in either the first stage or the second stage includes:

[0205] Each training data pair corresponding to this stage is input into the quality analysis model to be trained in this stage to obtain the quality prediction result of each training data pair; the quality prediction result of each training data pair is used to characterize: the degree of matching between the text and labels in the first training data included in the training data pair, compared with the degree of matching between the text and labels in the second training data included in the training data pair.

[0206] Identify training data pairs whose quality prediction results and quality labels do not meet the matching conditions.

[0207] Based on the difference between the quality prediction results and quality labels of the determined training data pairs, the model parameters of the specified layer of the quality ranking model to be trained in this stage are updated through backpropagation.

[0208] Optionally, the calling unit includes:

[0209] The calling subunit is used to call the target large language model based on each current target text and the annotation prompt words containing reference data of each current target text, so that the target large language model can perform label annotation on each current target text based on the annotation prompt words and analyze the annotation reasons to obtain the optimized candidate labels and label annotation reasons for each target text;

[0210] Accordingly, for each target text, if the selected reference text has a labeling reason, the reference data of the target text that contains the reference text also includes the labeling reason of the reference text.

[0211] Optionally, the identification module includes:

[0212] The first sorting unit is used to sort the texts according to the matching degree from high to low based on the quality prediction results of each sample, and obtain the first sorting result.

[0213] From the first sorting result, a predetermined proportion of texts ranked lower are selected to obtain the target texts whose labeling quality meets the optimization criteria; or...

[0214] The identification module includes:

[0215] The second sorting unit is used to sort the texts in each text group obtained after grouping the texts, based on the quality prediction results of each sample, according to the sorting method of matching degree from high to low, to obtain a second sorting result, and select a predetermined proportion of texts that are sorted lower from the second sorting result to obtain the target texts corresponding to the text group whose labeling quality meets the conditions to be optimized.

[0216] This application also provides an electronic device, such as... Figure 8 As shown, it includes:

[0217] Memory 801 is used to store computer programs;

[0218] The processor 802, when executing the program stored in the memory 801, implements any of the above-described text labeling methods.

[0219] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 802, the communication interface, and the memory 801 communicating with each other via the communication bus.

[0220] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0221] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0222] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0223] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0224] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described text tagging method.

[0225] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the text labeling methods described in the above embodiments.

[0226] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0227] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0228] The various embodiments in this specification are described in a related manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0229] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A label annotation method of text, characterized by, The method comprises the following steps: Based on the labeling model, the texts to be labeled are labeled and the candidate labels of the texts are obtained; Based on the texts and the candidate labels, a plurality of sample pairs are constructed; each sample pair comprises first text data and second text data, and each text data in the first text data and the second text data comprises a text and a candidate label of the text; Analyze the quality prediction result of each sample pair; the quality prediction result of each sample pair is used to represent the matching degree of the text and the candidate label in the first text data included in the sample pair, and the matching degree of the text and the candidate label in the second text data included in the sample pair; wherein the way to analyze the quality prediction result of each sample pair is: based on a large language model or a pre-trained target quality analysis model for analysis; Based on the obtained quality prediction result, identify the target texts in the texts whose label labeling quality meets the optimization condition; Optimize the labels of the current target texts to obtain optimized candidate labels; After label optimization, based on the current candidate labels of the texts, determine the label labeling results of the texts.

2. The method of claim 1, wherein, The method comprises the following steps: Output a predetermined proportion of texts and corresponding candidate labels in the texts to indicate that the predetermined proportion of texts are manually inspected; In response to obtaining an unqualified inspection result, return to the step of analyzing the quality prediction result of each sample pair; otherwise, determine the current candidate labels of the texts as the label labeling results of the texts.

3. The method according to claim 1 or 2, characterized in that, The method comprises the following steps: For each target text, select at least one reference text that meets the similarity condition from the texts other than the current target texts to obtain reference data of the target text, which contains the selected reference text and the candidate label of the selected reference text; Based on the current target texts and the labeling prompt words containing the reference data of each target text, call a target large language model to label the current each target text based on the labeling prompt words to obtain the candidate label of each optimized target text.

4. The method according to claim 1 or 2, characterized in that, The method comprises the following steps: Input each sample pair into a pre-trained target quality analysis model to obtain the quality prediction result of each sample pair; The target quality analysis model is trained based on each training data pair and the quality labeling label of each training data pair; each training data pair comprises first training data and second training data, and each training data in the first training data and the second training data comprises a training text and a label of the training text. The quality annotation label of each training data pair is used to represent a matching degree of the text and the label in the first training data included in the training data pair, and a target relationship of the matching degree of the text and the label in the second training data included in the training data pair, the target relationship including higher, same or lower.

5. The method of claim 4, wherein, The training manner of the target quality analysis model includes: The first training data set is used to perform first-stage training on the quality analysis model to be trained to obtain a quality analysis model trained in the first stage; wherein the first training data set includes a plurality of training data pairs constructed based on a predetermined public data set; The second training data set is used to perform second-stage training on the quality analysis model trained in the first stage to obtain a target quality analysis model trained; wherein the second training data set includes a plurality of training data pairs constructed based on text in a specified field, and the specified field is a field to which the text belongs.

6. The method of claim 5, wherein, The training process of any one of the first-stage training and the second-stage training includes: Each training data pair corresponding to the stage is input into the quality analysis model to be trained in the stage to obtain a quality prediction result of each training data pair; the quality prediction result of each training data pair is used to represent a matching degree of the text and the label in the first training data included in the training data pair, and a high-low condition of the matching degree of the text and the label in the second training data included in the training data pair; Determine the training data pair with the quality prediction result and the quality annotation label that does not meet the matching condition; Based on the difference between the quality prediction result and the quality annotation label of the determined training data pair, the model parameters of the specified layer of the quality analysis model to be trained in the stage are updated by means of back propagation.

7. The method of claim 3, wherein, The target large language model is called based on the current target text and the annotation prompt word containing the reference data of the current target text, so that the target large language model performs label annotation on the current target text based on the annotation prompt word to obtain the candidate label of the optimized target text, including: The target large language model is called based on the current target text and the annotation prompt word containing the reference data of the current target text, so that the target large language model performs label annotation and analyzes the annotation reason on the current target text based on the annotation prompt word to obtain the candidate label and the label annotation reason of the optimized target text; Correspondingly, for the current target text, if the selected reference text has a label annotation reason, the reference data of the target text containing the reference text also includes the label annotation reason of the reference text.

8. The method of claim 1 or 2, wherein, Based on the obtained quality prediction result, the target text in the text whose label annotation quality meets the to-be-optimized condition is identified, including: Based on the quality prediction result of each sample, the texts are sorted in a descending order of matching degree to obtain a first sorting result; From the first sorting result, a predetermined proportion of texts at the rear of the sorting is selected to obtain each target text in the plurality of texts whose label annotation quality meets the to-be-optimized condition; or The identifying, based on the obtained quality prediction result, of each target text in the plurality of texts whose label annotation quality meets the to-be-optimized condition includes: For each text group obtained after the grouping of the plurality of texts, based on the quality prediction result of each sample, the texts in the text group are sorted in a descending order of matching degree to obtain a second sorting result, and a predetermined proportion of texts at the rear of the sorting is selected from the second sorting result to obtain target texts corresponding to the text group and meeting the to-be-optimized condition in terms of label annotation quality.

9. A label annotation apparatus of text, characterized by, Comprise: The annotation module is configured to perform label annotation on the plurality of texts based on the annotation model to obtain candidate labels of the plurality of texts. The construction module is configured to construct a plurality of sample pairs based on the plurality of texts and the candidate labels; each sample pair includes first text data and second text data, and each text data in the first text data and the second text data includes a text and a candidate label of the text. The analysis module is configured to analyze the quality prediction result of each sample pair; the quality prediction result of each sample pair is used to represent the matching degree of the text and the candidate label in the first text data included in the sample pair, and the high-low situation of the matching degree of the text and the candidate label in the second text data included in the sample pair; wherein the analysis of the quality prediction result of each sample pair is based on a large language model or a pre-trained target quality analysis model. The identification module is configured to identify, based on the obtained quality prediction result, each target text in the plurality of texts whose label annotation quality meets the to-be-optimized condition. The optimization module is configured to perform label optimization on the current target texts to obtain optimized candidate labels. The determination module is configured to determine, after the label optimization, a label annotation result of the plurality of texts based on the current candidate labels of the plurality of texts.

10. An electronic device, comprising: Comprise: The memory is configured to store a computer program; The processor is configured to execute the program stored in the memory to implement the method of any one of claims 1-8.

11. A computer readable storage medium, characterized in that, The computer program is stored in the computer-readable storage medium, and the computer program is executed by the processor to implement the method of any one of claims 1-8.

Images