Large model data labeling method and device, equipment, medium and product

Abstract

The invention discloses a large model data annotation method and device, equipment, a medium and a product, and aims to inspect annotation consistency of a large model from different angles by performing grammar transformation on an input text and a cue word of a to-be-classified task to generate diversified large model input so as to more accurately assess an assessment result of the model. Multiple input versions are labeled through multiple pre-training language models, so that the understanding and labeling capability of a large model on input is evaluated more comprehensively. By acquiring the confidence score of each label, the credibility of different large models to different labeling results can be reflected, and the decision process of the model is more transparent by the quantitative index mode, so that the reliability of the pre-training language model in a data labeling task is improved, and the trust of a user to the labeling result of the pre-training language model is enhanced.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to large model data annotation methods, devices, equipment, media and products. Background Art

[0002] In the field of natural language processing, large language models are widely used in classification tasks such as sentiment analysis, fake news detection, bias identification, medical diagnosis, and legal document inference. High-quality data annotation is a key prerequisite for training accurate and reliable large models.

[0003] In related technologies, using large language models for data labeling can theoretically improve labeling efficiency and reduce labor costs. However, its reliability is low in some areas. For example, when encountering vague, ambiguous or rare text expressions, large language models may give inaccurate labeling.

[0004] Therefore, how to improve the reliability of large language models in data labeling tasks is an urgent problem that needs to be solved. Summary of the invention

[0005] The present application provides a large-model data annotation method, apparatus, device, medium and product to at least solve the problem of low reliability of pre-trained language models in data annotation tasks in related technologies.

[0006] The present application provides a large model data annotation method, the method comprising: Performing grammatical transformation on the input text and prompt words of the classification task respectively, and obtaining a plurality of target input texts and target prompt words after the grammatical transformation; Inputting the multiple target input texts and target prompt words into multiple pre-trained language models for text annotation, and obtaining multiple predicted labels output by the multiple pre-trained language models; Obtain weights of multiple target input texts, weights of multiple pre-trained language models, and probability thresholds of multiple labels; Calculating weighted confidence scores of the multiple labels based on the multiple predicted labels of the task to be classified, the weights of the multiple target input texts, and the weights of the multiple pre-trained language models; Determine the target labeling result of the task to be classified according to the weighted confidence scores of the multiple tags and the probability thresholds of the multiple tags.

[0007] The present application also provides a large model data annotation device, the device comprising: A transformation module, used to perform grammatical transformation on the input text and prompt words of the classification task, and obtain multiple target input texts and target prompt words after grammatical transformation; A labeling module, used for inputting the multiple target input texts and target prompt words into multiple pre-trained language models for text labeling, and obtaining multiple predicted labels output by the multiple pre-trained language models; An acquisition module, used to obtain weights of multiple target input texts, weights of multiple pre-trained language models, and probability thresholds of multiple tags; A calculation module, configured to calculate weighted confidence scores of a plurality of labels based on a plurality of predicted labels of the task to be classified, weights of the plurality of target input texts, and weights of the plurality of pre-trained language models; A determination module is used to determine the target labeling result of the task to be classified according to the weighted confidence scores of the multiple labels and the probability thresholds of the multiple labels.

[0008] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned large model data labeling methods when executing the computer program.

[0009] The present application also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned large model data labeling methods are implemented.

[0010] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned large model data labeling methods when executed by a processor.

[0011] Through this application, the input text and prompt words of the classification task are grammatically transformed respectively, and multiple target input texts and target prompt words after grammatical transformation are obtained. By performing grammatical transformation on the input text and prompt words of the classification task, a variety of large model inputs are generated, and the annotation consistency of the large model is examined from different angles, so as to more accurately evaluate the evaluation results of the model. Multiple target input texts and target prompt words are input into multiple pre-trained language models for text annotation, and multiple predicted labels output by multiple pre-trained language models are obtained. Multiple input versions are annotated by multiple pre-trained language models, so as to more comprehensively evaluate the understanding and annotation ability of the large model for input. The weights of multiple target input texts, the weights of multiple pre-trained language models, and the probability thresholds of multiple labels are obtained, and the weighted confidence scores of multiple labels are calculated based on the multiple predicted labels of the task to be classified, the weights of multiple target input texts, and the weights of multiple pre-trained language models. According to the weighted confidence scores of multiple labels and the probability thresholds of multiple labels, the target annotation results of the task to be classified are determined. By obtaining the confidence score of each label, the degree of credibility of different large models for different labeling results can be reflected. This quantitative indicator method makes the decision-making process of the model more transparent, thereby improving the reliability of the pre-trained language model in data labeling tasks and enhancing users' trust in the labeling results of the pre-trained language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0013] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0014] Figure 1 A schematic diagram of a large model data annotation method provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a large model data annotation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0016] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0017] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0018] Terminology explanation: Big model: Artificial intelligence big model refers to a "large parameter" model trained using large-scale data and powerful computing power. These models are usually highly versatile and generalizable, and can be applied to natural language processing, image recognition, speech recognition and other fields. They can be divided into big language models, big visual models, big multimodal models, basic big models, etc.

[0019] Data labeling: Processing, classifying, labeling, and annotating raw data (such as images, text, audio, video, etc.) so that it can be understood and used by machine learning models. Data labeling through large models refers to the process of automatically labeling unlabeled data using large-scale pre-trained language models (such as GPT, BERT, etc.). Data labeling with large models uses their powerful understanding and generation capabilities to automatically add labels to data and improve efficiency, but attention must be paid to accuracy and domain adaptability.

[0020] Grammatical transformation: Grammatical transformation refers to the process of changing the way a sentence is expressed in natural language processing (NLP) to generate new sentences or phrases while keeping the original meaning unchanged. It is used in scenarios such as data enhancement and model robustness improvement, but attention should be paid to semantic consistency and language fluency.

[0021] Pre-trained language model: generally refers to designing language model training tasks based on a large-scale corpus (including language training materials such as sentences and paragraphs), training a large-scale neural network algorithm structure to learn and implement, and the final large-scale neural network algorithm structure and parameters are the pre-trained language model. Subsequent tasks can be based on this model to extract features or fine-tune tasks to achieve specific task purposes. Among them, the neural network algorithm structure trained by the pre-trained language model can be CNN, RNN, LSTM, etc., or it can be a model built with an attention network, such as transformer, BERT, GPT, Clip, etc., which is not limited in this application. Attention network refers to a network model that uses an attention mechanism for training. The model assigns different weights to each part of the input sequence, thereby extracting more important feature information from the input sequence, so that the model finally obtains a more accurate output. The idea of ​​pre-training is to first train a task to obtain a set of model parameters, then use the set of model parameters to initialize the network model parameters, and then use the initialized network model to train other tasks to obtain models adapted to other tasks. By pre-training on a large-scale corpus, the neural language representation model can learn powerful language representation capabilities and extract rich syntactic and semantic information from the text. In the embodiments of the present disclosure, the pre-trained language model may include but is not limited to GPT, BERT, etc.

[0022] Since large language models have low reliability in data annotation in certain fields, they may give inaccurate annotations when encountering vague, ambiguous or rare text expressions. For example, in medical diagnosis assistance, incorrect annotations may mislead doctors' judgments and threaten patients' health; in legal text analysis, inaccurate annotations may affect judicial justice.

[0023] In related technologies, indicators such as accuracy, confidence, and majority voting are usually used to evaluate the annotation results of large language models. The accuracy indicator evaluates the model performance by calculating the ratio of the number of samples with correct annotations to the total number of samples. For example, in a test set containing 100 text samples, if the model correctly annotates 80 samples, the accuracy of the model is 80%. The confidence evaluation method directly asks the model about the uncertainty of its own output. Some large language models provide relevant interfaces that can return the confidence score of their own annotation results. The higher the score, the more confident the model is in the result. The majority voting method annotates the same input through multiple large language models, and then selects the annotation result with the highest frequency as the final output. For example, in the sentiment analysis task, multiple large language models classify the sentiment of a piece of text respectively. If most models believe that the text has a positive sentiment, it is finally labeled as positive.

[0024] However, the above methods all have defects to varying degrees. For example, the accuracy index is too simple and cannot reflect the performance of the model in complex scenarios. It does not take into account the model's ability to label samples of different difficulty levels, as well as the type and distribution of labeling errors. The self-confidence evaluation method relies on the model's own judgment of uncertainty. However, the large model is essentially a neural network model. The confidence evaluation mechanism within the model is not always accurate and reliable, and the confidence score given may not match the reliability of the actual labeling. The majority voting method does not consider the confidence of a single model and the performance differences of different models in different scenarios. In practical applications, different large language models may perform very differently when processing specific fields or specific types of text. Simple majority voting may mask the advantages of certain models in specific scenarios, resulting in a decrease in overall labeling accuracy. In summary, there is still a lack of an evaluation method that can further improve the labeling quality of large models.

[0025] Based on the above problems, an embodiment of the present application provides a large model data labeling method, and describes the method in detail in conjunction with the execution process of the large model data labeling method.

[0026] Reference Figure 1 As shown, the large model data labeling method provided by the embodiment of the present invention includes the following steps: S11, performing grammatical transformation on the input text and prompt words of the classification task respectively, and obtaining a plurality of target input texts and target prompt words after the grammatical transformation.

[0027] The input text refers to the original text content provided by the user or system to the pre-trained language model for some processing or analysis. It is usually the input data of the model and can be any type of natural language text, such as sentences, paragraphs, or entire documents. The prompt word is used to guide the pre-trained language model to generate specific outputs or perform specific tasks. It helps the pre-trained language model understand the user's needs by providing context or questions.

[0028] Specifically, since grammatical transformation reorganizes the structure of a sentence or text while keeping its semantics unchanged or basically unchanged, grammatical transformation is performed on the input text and prompt words of the classification task respectively to obtain multiple target input texts and target prompt words after grammatical transformation.

[0029] For example, the input text of the task to be classified is: "Xiao Ming is very happy today.", and the prompt word is "Please help me determine whether the above text expresses positive emotions?". Through different semantic transformation methods, multiple input texts and multiple prompt words after grammatical transformation are obtained.

[0030] In some embodiments, the above step S11 (determining the target labeling result of the task to be classified according to the weighted confidence scores of the multiple labels and the probability thresholds of the multiple labels) can be implemented by the following steps: (1) Performing grammatical transformation on the input text based on a first preset manner to obtain a plurality of target input texts after the grammatical transformation.

[0031] Optionally, the above step (1) can be implemented as follows: Performing voice transformation on the input text to obtain a first input text after grammatical transformation; Performing sentence transformation on the input text to obtain a second input text after grammatical transformation; Synonym replacement is performed on the input text to obtain a third input text after grammatical transformation.

[0032] Among them, voice transformation includes switching the voice of the input text between active voice and passive voice. Sentence transformation includes converting the positive phrase in the input text into two negative phrases while keeping the original meaning unchanged. Synonym replacement is to replace the keywords of the input text with synonyms to generate different expressions with the same semantics.

[0033] Specifically, the first input text is obtained by converting the active voice of the input text into the passive voice, or converting the passive voice of the input text into the active voice; the second input text is obtained by converting the affirmative phrase of the input text into two negative phrases, or converting the double negative phrase of the input text into an affirmative phrase; the third input text is obtained by replacing the keywords of the input text with synonyms.

[0034] For example, "Xiao Ming wrote a letter. He sent the letter to his friend." can be converted to "A letter was written by Xiao Ming. The letter was sent to his friend." This conversion can test whether the pre-trained language model can maintain the correct understanding of semantics when the sentence structure changes.

[0035] For example, "He will definitely come to the party." is transformed into "It is impossible for him not to come to the party." This grammatical transformation helps to test the model's sensitivity to semantic changes.

[0036] For example, "She likes reading novels very much, especially those full of suspense and thrilling plots." becomes "She particularly loves reading stories, especially those full of suspense and exciting plots." In this way, we can examine whether the model focuses on semantic content rather than superficial lexical differences.

[0037] Furthermore, the above step (1) can also be implemented in the following way: Adjusting the word order of the input text to obtain a fourth input text after grammatical transformation; The input text is expanded or simplified to obtain a fifth input text after grammatical transformation.

[0038] Specifically, by changing the order of words or phrases in the input text, sentences with similar semantics but different grammatical structures are generated. The complexity of the sentence is changed by adding or removing some modifiers while maintaining the core semantics.

[0039] Through different grammatical transformations, diverse input texts can be generated, or multiple question forms can be generated to help the pre-trained language model better understand the user's questions.

[0040] (2) Performing grammatical transformation on the prompt word based on a second preset method to obtain a target prompt word after the grammatical transformation.

[0041] Optionally, the above step (2) can be implemented as follows: The prompt word is subjected to sentence transformation to obtain a target prompt word after grammatical transformation.

[0042] Specifically, by performing double negation conversion on the affirmative phrase in the prompt word, the target prompt word after grammatical transformation can be obtained.

[0043] For example, “Please help me determine whether the following text expresses positive emotions?” can be transformed into “Please help me determine whether the following text does not express negative emotions.” This grammatical transformation helps test the model’s sensitivity to semantic transformations of reasoning prompts.

[0044] By performing grammatical transformation on the input text and prompt words of the classification task, we can generate diverse large model inputs and examine the annotation consistency of the large model from different angles, so as to more accurately evaluate the evaluation results of the model.

[0045] S12: Input the multiple target input texts and target prompt words into multiple pre-trained language models for text annotation, and obtain multiple predicted labels output by the multiple pre-trained language models.

[0046] Specifically, the original input text and the original prompt word, as well as the multiple target input texts and the target prompt words after grammatical transformation are input into multiple pre-trained language models for labeling, and multiple predicted labels output by the multiple pre-trained language models are obtained.

[0047] For example, taking the sentiment classification task as an example, assuming that the possible annotation results (predicted labels) include three categories: positive emotions, negative emotions, and neutral emotions. If a pre-trained language model labels multiple inputs (taking 7 inputs as an example), 5 of the annotation results are positive emotions and 2 are negative emotions. Then the probability distribution of the label type being positive emotion is 5 / 7, and the probability distribution of the label type being negative emotion is 2 / 7.

[0048] S13, obtaining weights of multiple target input texts, weights of multiple pre-trained language models, and probability thresholds of multiple labels.

[0049] Specifically, in the process of annotation quality assessment, three types of parameters are involved, namely Iweight, Mweight, and kL. Among them, Iweight represents the weight assigned to a given input text, Mweight represents the weight assigned to a given pre-trained language model M, and kL represents the probability threshold of each label. For specific tasks, users with historical experience can set the weights of the three types of parameters based on experience, but in general, the weights are inevitably affected by personal bias, limited experience, and cognitive limitations due to human subjective determination. Different people may give very different weights, and these weights are not necessarily based on the true characteristics of the data and the requirements of the model. Different classification tasks and data sets have unique characteristics and require different weight configurations to achieve optimal performance. There are also complex interactions between the different weight factors of the three types of parameters. For example, the weight coefficients of different grammatical transformations have different effects on different large models, and different large models also perform differently on different tasks. Therefore, a way is needed to determine that the weight values ​​of the above three parameters are within a reasonable range.

[0050] For classification tasks in most scenarios, users already have a certain number of training samples with correctly labeled results. Based on this specific task and this specific training sample, the optimal values ​​of the above three types of weight coefficients can be determined through heuristic learning methods.

[0051] Optionally, the above step S13 can be implemented in the following manner: Initialize the candidate solution population.

[0052] The candidate solution population includes initial weights of multiple target input texts, initial weights of multiple pre-trained language models, and initial probability thresholds of multiple labels; Evaluate the fitness of each candidate solution based on the accuracy function; Perform crossover and mutation operations based on the fitness of each candidate solution to generate a new candidate solution population; Through multiple iterations, the candidate solution population is gradually optimized until a preset number of iterations is reached, and the weights of multiple target input texts, the weights of multiple pre-trained language models, and the probability thresholds of multiple labels are determined.

[0053] Specifically, the candidate solution population is initialized, and multiple random assignments are performed to generate a candidate solution population including Iweight, Mweight, and kL, and the range of these values ​​is between (0, 1). Then, the accuracy function is used to evaluate the fitness of each candidate solution, and the population mutation and crossover process is iterated through the distribution of the initial population fitness. During crossover, new candidate solutions are generated by combining some of the values ​​of the randomly exchanged candidate solutions; during mutation, new candidate solutions are generated by randomly changing some of the values ​​of the candidate solutions. If the optimal fitness of the new candidate solution is better than the optimal fitness of the current population, the current optimal fitness is replaced. This process continues for several iterations or until convergence, and finally returns the optimized parameters. During the optimization process, the parameters must meet specific constraints to ensure that the weights of the pre-trained language model, the weights of the input text, and the probability threshold are within a reasonable range.

[0054] In addition, for big data samples, after obtaining the weights of multiple target input texts, the weights of multiple pre-trained language models, and the probability thresholds of multiple labels, these weight coefficients and probability thresholds can be combined to complete the labeling of the remaining samples.

[0055] Among them, the accuracy function is expressed as: The accuracy function is used to minimize the loss function to maximize the accuracy, where the loss function is expressed by the following formula: .

[0056] in, represents the loss value between the predicted label and the true label, n represents the number of samples with the true label, and y pred is the predicted label, y true is the annotated true label.

[0057] Through heuristic algorithms, parameter weights are adjusted based on objective data and established optimization strategies, which is not interfered by subjective factors and can provide more objective and reliable weights, thereby ensuring the stability and reliability of evaluation performance.

[0058] S14. Calculate weighted confidence scores of the multiple labels based on the multiple predicted labels of the task to be classified, the weights of the multiple target input texts, and the weights of the multiple pre-trained language models.

[0059] In some embodiments, the above step S14 (calculating weighted confidence scores of multiple labels based on the multiple predicted labels of the task to be classified, the weights of the multiple target input texts, and the weights of the multiple pre-trained language models) can be implemented by the following steps: A. Based on the multiple predicted labels of the task to be classified and the weights of the multiple target input texts, calculate the confidence score of the target pre-trained language model for the target label.

[0060] Optionally, the above step A can be implemented in the following manner: A1. Obtain the preset weight of the target prediction label.

[0061] Get the target prediction label assigned by the target pre-trained language model to the target input text; Determine whether the target prediction label is consistent with the target label; If the target prediction label is consistent with the target label, determining the preset weight of the target prediction label to be a first value; If the target prediction label is inconsistent with the target label, the preset weight of the target prediction label is determined to be a second value.

[0062] The target pre-trained language model is any one of the multiple pre-trained language models. The target label is any label that the pre-trained language model may include when annotating the target input text. For example, taking the sentiment classification task as an example, the input text is: "Xiao Ming is very happy today.", and the prompt is "Please help me judge whether the above text expresses positive emotions?" In this case, the target labels may include: negative emotions, positive emotions, and neutral emotions.

[0063] Specifically, obtain the target prediction label assigned by the target pre-trained language model to the target input text, and determine whether the target prediction label is consistent with the target label. If the target prediction label is consistent with the target label, determine the preset weight of the target prediction label to be 1; if the target prediction label is inconsistent with the target label, determine the preset weight of the target prediction label to be 0.

[0064] A2. Calculate the confidence score of the target pre-trained language model for the target label based on the preset weight of the target prediction label and the weights of the multiple target input texts.

[0065] Specifically, according to the preset weight of the target prediction label and the weights of multiple target input texts, the confidence score of the target pre-trained language model for the target label is calculated.

[0066] Optionally, the above step A2 can be calculated by the following formula: .

[0067] in, Represents the confidence score of the target pre-trained language model M for the target label L; Indicates the weight assigned to the target input text, and its value range is 0-1. The sum of the weights of multiple target input texts is 1; Represents the target pre-trained language model M for the target input text l assigned labels; Representation Tags The preset weight of When it is equal to L, The value of is 1; when When it is not equal to L, The value of is 0.

[0068] B. Based on the confidence score of the target pre-trained language model for the target label and the weights of the multiple pre-trained language models, calculate the weighted confidence scores of the multiple pre-trained language models for the target label.

[0069] Optionally, the above step B can be implemented in the following manner: B1. Calculate the confidence scores of each pre-trained language model for each label; B2. Calculate the weighted confidence scores of multiple pre-trained language models for the target label based on the weights of each pre-trained language model and the confidence scores of each pre-trained language model for each label.

[0070] Specifically, the confidence scores of each pre-trained language model for each label are calculated respectively, and then the weighted confidence scores of multiple pre-trained language models for the target label are calculated according to the weights of each pre-trained language model and the confidence scores of each pre-trained language model for each label.

[0071] Optionally, the above step B can be calculated by the following formula: .

[0072] in, represents the weighted confidence scores of multiple pre-trained language models (i.e., all pre-trained language models in the embodiment of the present disclosure) for the target label L; Represents the sum of the weights of multiple pre-trained language models, and the value is 1; Represents the weight assigned to the target pre-trained language model M, and its value range is 0-1.

[0073] S15. Determine a target labeling result of the task to be classified according to the weighted confidence scores of the multiple tags and the probability thresholds of the multiple tags.

[0074] In some embodiments, if the task to be classified is a single classification task, the above step S15 (determining the target labeling result of the task to be classified according to the weighted confidence scores of the multiple labels and the probability thresholds of the multiple labels) can be implemented as follows: Determine whether the weighted confidence score of each label is greater than the probability threshold of each label; If the weighted confidence score of the target label among the multiple labels is greater than the probability threshold of the target label, the target label is determined to be the target labeling result of the task to be classified.

[0075] Among them, the probability threshold corresponding to each label is different.

[0076] Specifically, the weighted confidence score of each label is judged according to the set probability threshold to determine the final label. That is, it is determined whether the weighted confidence score of each label is greater than the probability threshold of each label. If there is a target label among multiple labels that meets its corresponding threshold condition, then the target label is determined to be the target labeling result of the task to be classified.

[0077] If the weighted confidence scores of at least two tags among the multiple tags are greater than the probability threshold of the at least two tags, then calculating the relative scores of the at least two tags; The relative scores of the at least two tags are sorted, and the tag with the highest relative score among the at least two tags is determined as the target labeling result of the task to be classified.

[0078] Specifically, if at least two tags among the multiple tags meet their corresponding threshold conditions, the relative scores of the at least two tags are calculated according to the following formula: .

[0079] in, Represents the relative score of any label L; Represents the probability threshold of any label L.

[0080] In some embodiments, if the task to be classified is a multi-classification task, the above step S15 (determining the target labeling result of the task to be classified according to the weighted confidence scores of the multiple tags and the probability thresholds of the multiple tags) can be implemented as follows: Determine whether the weighted confidence score of each label is greater than the probability threshold of each label; If the weighted confidence scores of at least two tags among the multiple tags are greater than the probability threshold of the at least two tags, then calculating the relative scores of the at least two tags; The relative scores of the at least two labels are sorted, and the top N labels with the highest relative scores among the at least two labels are determined as the target labeling results of the task to be classified.

[0081] Wherein, N is greater than or equal to 2.

[0082] Specifically, for a multi-classification task, if multiple labels meet the threshold condition, the labels corresponding to the multiple classification tasks are determined according to the score ranking of the relative scores.

[0083] This method uses a multi-task scenario and is not only applicable to a single labeling task of a single pre-trained language model, but can also be performed in multiple labeling task scenarios of multiple pre-trained language models. By reasonably allocating weights between labels and comprehensively considering the outputs of multiple models, the weighted confidence scores of multiple labels are calculated, and the final label label is determined. This method can make full use of the evaluation indicator (i.e., probability threshold) to improve the accuracy of the labeling results.

[0084] The large model data annotation method provided by the embodiment of the present disclosure performs grammatical transformation on the input text and prompt words of the classification task respectively, and obtains multiple target input texts and target prompt words after grammatical transformation. By performing grammatical transformation on the input text and prompt words of the classification task, a diversified large model input is generated, and the annotation consistency of the large model is examined from different angles, so as to more accurately evaluate the evaluation result of the model. Multiple target input texts and target prompt words are input into multiple pre-trained language models for text annotation, and multiple predicted labels output by multiple pre-trained language models are obtained. Multiple input versions are annotated by multiple pre-trained language models, so as to more comprehensively evaluate the understanding and annotation ability of the large model for the input. The weights of multiple target input texts, the weights of multiple pre-trained language models, and the probability thresholds of multiple labels are obtained, and the weighted confidence scores of multiple labels are calculated based on the multiple predicted labels of the classification task, the weights of multiple target input texts, and the weights of multiple pre-trained language models. According to the weighted confidence scores of multiple labels and the probability thresholds of multiple labels, the target annotation result of the classification task is determined. By obtaining the confidence score of each label, the degree of credibility of different large models for different labeling results can be reflected. This quantitative indicator method makes the decision-making process of the model more transparent, thereby improving the reliability of the pre-trained language model in data labeling tasks and enhancing users' trust in the labeling results of the pre-trained language model.

[0085] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0086] The embodiment of the present application also provides a large model data annotation device. The omitted virtual device claim content should be described in detail in the specification and correspond one-to-one with the method claim.

[0087] Figure 2 A schematic diagram of the structure of a large model data annotation device 200 provided by the present disclosure is shown in FIG. Figure 2 As shown, the device of this embodiment includes: a transformation module 210, a labeling module 220, an acquisition module 230, a calculation module 240 and a determination module 250, wherein: The transformation module 210 is used to perform grammatical transformation on the input text and prompt words of the classification task to obtain a plurality of target input texts and target prompt words after grammatical transformation; Annotation module 220, used for inputting the multiple target input texts and target prompt words into multiple pre-trained language models for text annotation, and obtaining multiple predicted labels output by the multiple pre-trained language models; An acquisition module 230, used to acquire weights of multiple target input texts, weights of multiple pre-trained language models, and probability thresholds of multiple tags; A calculation module 240, configured to calculate weighted confidence scores of a plurality of labels based on a plurality of predicted labels of the task to be classified, weights of the plurality of target input texts, and weights of the plurality of pre-trained language models; The determination module 250 is used to determine the target labeling result of the task to be classified according to the weighted confidence scores of the multiple tags and the probability thresholds of the multiple tags.

[0088] As an optional implementation of the embodiment of the present disclosure, the transformation module 210 includes: A first grammar transformation unit, configured to perform grammar transformation on the input text based on a first preset method, and obtain a plurality of target input texts after grammar transformation; The second grammatical transformation unit is used to perform grammatical transformation on the prompt word based on a second preset method to obtain a target prompt word after the grammatical transformation.

[0089] As an optional implementation manner of the embodiment of the present disclosure, the first grammar transformation unit is specifically used to: Performing voice transformation on the input text to obtain a first input text after grammatical transformation; Performing sentence transformation on the input text to obtain a second input text after grammatical transformation; Synonym replacement is performed on the input text to obtain a third input text after grammatical transformation.

[0090] As an optional implementation manner of the embodiment of the present disclosure, the second grammar transformation unit is specifically used to: The prompt word is subjected to sentence transformation to obtain a target prompt word after grammatical transformation.

[0091] As an optional implementation of the embodiment of the present disclosure, the calculation module 240 includes: A confidence score calculation unit, configured to calculate a confidence score of a target pre-trained language model for a target label based on the weights of the plurality of predicted labels of the task to be classified and the plurality of target input texts; A weighted confidence score calculation unit is used to calculate the weighted confidence scores of the multiple pre-trained language models for the target label based on the confidence score of the target pre-trained language model for the target label and the weights of the multiple pre-trained language models.

[0092] As an optional implementation of the embodiment of the present disclosure, the confidence score calculation unit includes: An acquisition subunit is used to obtain a preset weight of a target prediction label; A calculation subunit is used to calculate the confidence score of the target pre-trained language model for the target label according to the preset weight of the target prediction label and the weights of the multiple target input texts.

[0093] As an optional implementation of the embodiment of the present disclosure, the acquisition subunit is specifically used to: Get the target prediction label assigned by the target pre-trained language model to the target input text; Determine whether the target prediction label is consistent with the target label; If the target prediction label is consistent with the target label, determining the preset weight of the target prediction label to be a first value; If the target prediction label is inconsistent with the target label, the preset weight of the target prediction label is determined to be a second value.

[0094] As an optional implementation of the embodiment of the present disclosure, the weighted confidence score calculation unit is specifically used to: Calculate the confidence scores of each pre-trained language model for each label respectively; According to the weights of each pre-trained language model and the confidence scores of each pre-trained language model for each label, the weighted confidence scores of multiple pre-trained language models for the target label are calculated.

[0095] As an optional implementation of the embodiment of the present disclosure, if the task to be classified is a single classification task, the determination module 250 includes: A judgment unit, used to judge whether the weighted confidence score of each tag is greater than the probability threshold of each tag; A comparing unit is used to determine that the target tag is a target labeling result of the task to be classified if the weighted confidence score of the target tag among the multiple tags is greater than the probability threshold of the target tag.

[0096] As an optional implementation of the embodiment of the present disclosure, the judgment unit is further specifically configured to: If the weighted confidence scores of at least two tags among the multiple tags are greater than the probability threshold of the at least two tags, then calculating the relative scores of the at least two tags; The relative scores of the at least two tags are sorted, and the tag with the highest relative score among the at least two tags is determined as the target labeling result of the task to be classified.

[0097] For the description of the features in the embodiment corresponding to the large model data labeling device 200, reference can be made to the relevant description of the embodiment corresponding to the large model data labeling method, which will not be repeated here.

[0098] The large model data annotation device provided by the embodiment of the present disclosure performs grammatical transformation on the input text and prompt words of the classification task respectively, and obtains multiple target input texts and target prompt words after grammatical transformation. By performing grammatical transformation on the input text and prompt words of the classification task, a diversified large model input is generated, and the annotation consistency of the large model is examined from different angles, so as to more accurately evaluate the evaluation result of the model. Multiple target input texts and target prompt words are input into multiple pre-trained language models for text annotation, and multiple predicted labels output by multiple pre-trained language models are obtained. Multiple input versions are annotated by multiple pre-trained language models, so as to more comprehensively evaluate the understanding and annotation ability of the large model for input. The weights of multiple target input texts, the weights of multiple pre-trained language models, and the probability thresholds of multiple labels are obtained, and the weighted confidence scores of multiple labels are calculated based on the multiple predicted labels of the classification task, the weights of multiple target input texts, and the weights of multiple pre-trained language models. According to the weighted confidence scores of multiple labels and the probability thresholds of multiple labels, the target annotation result of the classification task is determined. By obtaining the confidence score of each label, the degree of credibility of different large models for different labeling results can be reflected. This quantitative indicator method makes the decision-making process of the model more transparent, thereby improving the reliability of the pre-trained language model in data labeling tasks and enhancing users' trust in the labeling results of the pre-trained language model.

[0099] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned large model data labeling method embodiments.

[0100] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned large model data labeling method embodiments when running.

[0101] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0102] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned large model data labeling method embodiments are implemented.

[0103] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned large model data labeling method embodiments are implemented.

[0104] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0105] The above is a detailed introduction to a large model data annotation method provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A large model data annotation method, characterized in that: The method comprises: Performing grammatical transformation on the input text and prompt words of the classification task respectively, and obtaining a plurality of target input texts and target prompt words after the grammatical transformation; Inputting the multiple target input texts and target prompt words into multiple pre-trained language models for text annotation, and obtaining multiple predicted labels output by the multiple pre-trained language models; Obtain weights of multiple target input texts, weights of multiple pre-trained language models, and probability thresholds of multiple labels; Calculating weighted confidence scores of the multiple labels based on the multiple predicted labels of the task to be classified, the weights of the multiple target input texts, and the weights of the multiple pre-trained language models; Determine the target labeling result of the task to be classified according to the weighted confidence scores of the multiple tags and the probability thresholds of the multiple tags.

2. The large model data annotation method according to claim 1, characterized in that: The step of performing grammatical transformation on the input text and prompt words of the classification task to be processed, and obtaining a plurality of target input texts and target prompt words after the grammatical transformation, comprises: Performing grammatical transformation on the input text based on a first preset manner to obtain a plurality of target input texts after grammatical transformation; The prompt word is grammatically transformed based on a second preset manner to obtain a target prompt word after the grammatical transformation.

3. The large model data annotation method according to claim 2, characterized in that: The step of performing grammatical transformation on the input text based on a first preset manner to obtain a plurality of target input texts after grammatical transformation includes: Performing voice transformation on the input text to obtain a first input text after grammatical transformation; Performing sentence transformation on the input text to obtain a second input text after grammatical transformation; Synonym replacement is performed on the input text to obtain a third input text after grammatical transformation.

4. The large model data annotation method according to claim 2, characterized in that: The step of performing grammatical transformation on the prompt word based on the second preset method to obtain a target prompt word after grammatical transformation includes: The prompt word is subjected to sentence transformation to obtain a target prompt word after grammatical transformation.

5. The large model data annotation method according to claim 1, characterized in that: The step of calculating weighted confidence scores of the plurality of labels based on the plurality of predicted labels of the task to be classified, the weights of the plurality of target input texts, and the weights of the plurality of pre-trained language models comprises: Calculating a confidence score of a target pre-trained language model for a target label based on the multiple predicted labels of the task to be classified and the weights of the multiple target input texts; Based on the confidence score of the target pre-trained language model for the target label and the weights of the multiple pre-trained language models, weighted confidence scores of the multiple pre-trained language models for the target label are calculated.

6. The large model data annotation method according to claim 5, characterized in that: The step of calculating the confidence score of the target pre-trained language model for the target label based on the weights of the multiple predicted labels of the task to be classified and the multiple target input texts includes: Get the preset weight of the target prediction label; According to the preset weight of the target predicted label and the weights of the multiple target input texts, the confidence score of the target pre-trained language model for the target label is calculated.

7. The large model data annotation method according to claim 6, characterized in that: The step of obtaining a preset weight of a target prediction label includes: Get the target prediction label assigned by the target pre-trained language model to the target input text; Determine whether the target prediction label is consistent with the target label; If the target prediction label is consistent with the target label, determining the preset weight of the target prediction label to be a first value; If the target prediction label is inconsistent with the target label, the preset weight of the target prediction label is determined to be a second value.

8. The large model data annotation method according to claim 5, characterized in that: The calculating, based on the confidence score of the target pre-trained language model for the target label and the weights of the multiple pre-trained language models, weighted confidence scores of the multiple pre-trained language models for the target label comprises: Calculate the confidence scores of each pre-trained language model for each label respectively; According to the weights of each pre-trained language model and the confidence scores of each pre-trained language model for each label, the weighted confidence scores of multiple pre-trained language models for the target label are calculated.

9. The large model data annotation method according to claim 1, characterized in that: If the task to be classified is a single classification task, determining the target labeling result of the task to be classified according to the weighted confidence scores of the multiple labels and the probability thresholds of the multiple labels includes: Determine whether the weighted confidence score of each label is greater than the probability threshold of each label; If the weighted confidence score of the target label among the multiple labels is greater than the probability threshold of the target label, the target label is determined to be the target labeling result of the task to be classified.

10. The large model data labeling method according to claim 9, characterized in that: The step of judging whether the weighted confidence score of each predicted label is greater than the probability threshold of each label further includes: If the weighted confidence scores of at least two tags among the multiple tags are greater than the probability threshold of the at least two tags, then calculating the relative scores of the at least two tags; The relative scores of the at least two tags are sorted, and the tag with the highest relative score among the at least two tags is determined as the target labeling result of the task to be classified.

11. The large model data annotation method according to claim 1, characterized in that: The step of obtaining weights of multiple target input texts, weights of multiple pre-trained language models, and probability thresholds of multiple labels includes: Initializing a candidate solution population; the candidate solution population includes initial weights of multiple target input texts, initial weights of multiple pre-trained language models, and initial probability thresholds of multiple labels; Evaluate the fitness of each candidate solution based on the accuracy function; Perform crossover and mutation operations based on the fitness of each candidate solution to generate a new candidate solution population; Through multiple iterations, the candidate solution population is gradually optimized until a preset number of iterations is reached, and weights of multiple target input texts, weights of multiple pre-trained language models, and probability thresholds of multiple labels are determined.

12. A large model data annotation device, characterized in that: The device comprises: A transformation module, used to perform grammatical transformation on the input text and prompt words of the classification task, and obtain multiple target input texts and target prompt words after grammatical transformation; A labeling module, used for inputting the multiple target input texts and target prompt words into multiple pre-trained language models for text labeling, and obtaining multiple predicted labels output by the multiple pre-trained language models; An acquisition module, used to obtain weights of multiple target input texts, weights of multiple pre-trained language models, and probability thresholds of multiple tags; A calculation module, configured to calculate weighted confidence scores of a plurality of labels based on a plurality of predicted labels of the task to be classified, weights of the plurality of target input texts, and weights of the plurality of pre-trained language models; A determination module is used to determine the target labeling result of the task to be classified according to the weighted confidence scores of the multiple labels and the probability thresholds of the multiple labels.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the large model data labeling method as claimed in any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the large model data labeling method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the large model data labeling method as described in any one of claims 1 to 11 are implemented.

Images