Label generation method and device, electronic equipment and computer storage medium

Abstract

This application provides a label generation method, apparatus, electronic device, and computer storage medium. The method includes: acquiring a labeled sample dataset and an unlabeled sample dataset, wherein the labeled sample dataset includes multiple initial sample data and the true labels of each initial sample data, and the unlabeled sample dataset includes multiple unlabeled sample data; determining multiple sample quality assessment scores corresponding to the multiple initial sample data; filtering the multiple initial sample data based on the multiple sample quality assessment scores to obtain multiple candidate sample data; and generating a target label corresponding to each unlabeled sample data in the multiple unlabeled sample data, based on each unlabeled sample data and the multiple candidate sample data. The embodiments provided by this solution can improve the efficiency and accuracy of label generation.

Description

Technical Field

[0001] This application relates to the field of label processing technology, and in particular to a label generation method, apparatus, electronic device, and computer storage medium. Background Technology

[0002] Training large language models typically requires the model to learn the mapping relationship between inputs and outputs from labeled data. A large amount of high-quality, real-world labeled data helps the model identify patterns between input text and corresponding output labels, thereby adjusting the model's parameters. However, label acquisition methods in related technologies are usually based on manual labeling or semi-supervised learning. These methods suffer from drawbacks such as high cost, long timeframes, low label accuracy, and susceptibility to human bias, leading to low training efficiency for large language models. Summary of the Invention

[0003] This application provides a label generation method, apparatus, electronic device, and computer storage medium, which can improve label generation efficiency and label accuracy. The above technical solution is as follows:

[0004] In a first aspect, embodiments of this application provide a label generation method, the method comprising:

[0005] Obtain labeled sample datasets and unlabeled sample datasets. The labeled sample dataset includes multiple initial sample data and the true labels of each initial sample data. The unlabeled sample dataset includes multiple sample data to be labeled.

[0006] Determine the sample quality assessment scores corresponding to the above initial sample data;

[0007] Based on the above multiple sample quality assessment scores, the above multiple initial sample data are screened to obtain multiple candidate sample data;

[0008] For each of the above unlabeled sample data, a target label corresponding to each of the above unlabeled sample data and the above candidate sample data is generated.

[0009] In one possible implementation, determining the multiple sample quality assessment scores corresponding to the multiple initial sample data includes:

[0010] For each of the aforementioned initial sample data, determine the average similarity score between the aforementioned initial sample data and the aforementioned multiple initial sample data.

[0011] Determine the influence score of the initial sample data in the labeled sample dataset.

[0012] Determine the information increment score of the initial sample data in the labeled sample dataset;

[0013] Based on the above average similarity score, the above influence score, and the above information increment score, the sample quality assessment score corresponding to the above initial sample data is determined, and then multiple sample quality assessment scores corresponding to the above multiple initial sample data are obtained.

[0014] In one possible implementation, determining the average similarity score between the initial sample data and the plurality of initial sample data includes:

[0015] Determine the semantic vector corresponding to the initial sample data mentioned above;

[0016] Based on the semantic vectors corresponding to the initial sample data, multiple first similarity scores are generated between the initial sample data and the multiple initial sample data.

[0017] The average similarity score corresponding to the initial sample data is determined based on the multiple first similarity scores mentioned above.

[0018] In one possible implementation, determining the influence score of the initial sample data in the labeled sample dataset includes:

[0019] By using a pre-defined large language model to predict the labels of the labeled sample dataset, the first label prediction probability of the true labels of all initial sample data in the labeled sample dataset output by the large language model is obtained.

[0020] The current initial sample data is removed from the labeled sample dataset to obtain a subset of labeled sample data;

[0021] By using the large language model described above to predict the labels of the labeled sample data subset, the second label prediction probability of the true labels of all initial sample data in the labeled sample data subset output by the large language model is obtained.

[0022] The influence score of the initial sample data in the labeled sample dataset is determined based on the predicted probabilities of the first label and the second label.

[0023] In one possible implementation, determining the information increment score of the initial sample data in the labeled sample dataset includes:

[0024] By using a pre-defined large language model to predict the labels of the labeled sample dataset, the first label prediction probability of the true labels of all initial sample data in the labeled sample dataset output by the large language model is obtained.

[0025] Based on the predicted probability of the first label, the entropy data of the initial sample data is determined.

[0026] Based on the predicted probability of the first label, determine multiple cross-entropy data between the initial sample data and the multiple initial sample data;

[0027] The information increment score corresponding to the initial sample data is determined based on the self-entropy data and the multiple cross-entropy data mentioned above.

[0028] In one possible implementation, determining the sample quality assessment score corresponding to the initial sample data based on the average similarity score, the influence score, and the information increment score includes:

[0029] Obtain the first weight corresponding to the average similarity score, the second weight corresponding to the influence score, and the third weight corresponding to the information increment score;

[0030] The sample quality assessment score corresponding to the initial sample data is determined based on the first weight, the second weight, the third weight, the average similarity score, the influence score, and the information increment score.

[0031] In one possible implementation, the aforementioned initial sample data and the aforementioned sample quality assessment scores correspond one-to-one. Based on the aforementioned multiple sample quality assessment scores, the aforementioned initial sample data are filtered to obtain multiple candidate sample data, including:

[0032] The initial sample data were sorted based on the quality assessment scores of the samples to obtain the sorting results;

[0033] The first preset number of initial sample data with the highest sample quality evaluation score in the above sorting results are determined as multiple candidate sample data.

[0034] In one possible implementation, the process of generating target labels corresponding to the unlabeled sample data based on the unlabeled sample data and the plurality of candidate sample data includes:

[0035] Determine multiple second similarity scores between each of the above-mentioned unlabeled sample data and the above-mentioned multiple candidate sample data;

[0036] Determine the word coverage scores between each of the above unannotated sample data and the above multiple candidate sample data;

[0037] Based on the aforementioned multiple second similarity scores and multiple word coverage scores, multiple corrected similarity scores are determined for the aforementioned multiple candidate sample data;

[0038] The target sample data is determined from the candidate sample data based on the above-mentioned multiple modified similarity scores;

[0039] The real labels corresponding to the target sample data are determined as the target labels corresponding to the sample data to be labeled.

[0040] In one possible implementation, determining the multiple word coverage scores between each of the unannotated sample data and the multiple candidate sample data includes:

[0041] The first set of words included in each of the above-mentioned sample data to be labeled is extracted using a preset part-of-speech recognition tool;

[0042] The second vocabulary set included in each candidate sample data in the above-mentioned multiple candidate sample data is extracted using the aforementioned preset part-of-speech recognition tool;

[0043] Based on the first vocabulary set and the second vocabulary set included in each candidate sample data, determine the vocabulary coverage scores between each unannotated sample data and the multiple candidate sample data.

[0044] In one possible implementation, the above method also includes:

[0045] Based on the target labels corresponding to the above sample data to be labeled, generate prompt information corresponding to the above sample data to be labeled.

[0046] The pre-set large language model is trained based on the above-mentioned unlabeled sample data and the above-mentioned prompt information.

[0047] Secondly, embodiments of this application provide a label generation apparatus, the apparatus comprising:

[0048] The acquisition module is used to acquire labeled sample datasets and unlabeled sample datasets. The labeled sample datasets include multiple initial sample data and the real labels of each initial sample data. The unlabeled sample datasets include multiple sample data to be labeled.

[0049] The determination module is used to determine the multiple sample quality assessment scores corresponding to the above multiple initial sample data;

[0050] The filtering module is used to filter the initial sample data based on the above multiple sample quality assessment scores to obtain multiple candidate sample data;

[0051] The first generation module is used to generate target labels corresponding to each of the multiple unlabeled sample data based on the multiple unlabeled sample data and the multiple candidate sample data.

[0052] In one possible implementation, the aforementioned determining module includes:

[0053] The first determining unit is used to determine the average similarity score between each initial sample data and the plurality of initial sample data for each initial sample data in the plurality of initial sample data;

[0054] The second determining unit is used to determine the influence score of the initial sample data in the labeled sample dataset.

[0055] The third determining unit is used to determine the information increment score of the initial sample data in the labeled sample dataset.

[0056] The fourth determining unit is used to determine the sample quality assessment score corresponding to the initial sample data based on the average similarity score, the influence score, and the information increment score, thereby obtaining multiple sample quality assessment scores corresponding to the multiple initial sample data.

[0057] In one possible implementation, the first determining unit includes:

[0058] The first determining subunit is used to determine the semantic vector corresponding to the aforementioned initial sample data;

[0059] A sub-unit is generated to generate multiple first similarity scores between the initial sample data and the multiple initial sample data based on the semantic vector corresponding to the initial sample data.

[0060] The second determining subunit is used to determine the average similarity score corresponding to the initial sample data based on the multiple first similarity scores.

[0061] In one possible implementation, the second determining unit includes:

[0062] The first prediction subunit is used to predict the labels of the labeled sample dataset by using a preset large language model, and to obtain the first label prediction probability of the true labels of all initial sample data in the labeled sample dataset output by the large language model.

[0063] The elimination sub-unit is used to remove the current initial sample data from the labeled sample dataset to obtain a subset of labeled sample data.

[0064] The second prediction subunit is used to predict the labels of the labeled sample data subset by using the large language model to obtain the second label prediction probability of the true labels of all initial sample data in the labeled sample data subset output by the large language model.

[0065] The third determining subunit is used to determine the influence score of the initial sample data in the labeled sample dataset based on the first label prediction probability and the second label prediction probability.

[0066] In one possible implementation, the third determining unit mentioned above includes:

[0067] The third prediction subunit is used to predict the labels of the labeled sample dataset by using a preset large language model, and to obtain the first label prediction probability of the true labels of all initial sample data in the labeled sample dataset output by the large language model.

[0068] The fourth determining subunit is used to determine the self-entropy data of the initial sample data based on the predicted probability of the first label.

[0069] The fifth determining subunit is used to determine multiple cross-entropy data between the initial sample data and the multiple initial sample data based on the predicted probability of the first label.

[0070] The sixth determining subunit is used to determine the information increment score corresponding to the initial sample data based on the self-entropy data and the multiple cross-entropy data.

[0071] In one possible implementation, the fourth determining unit mentioned above includes:

[0072] The sub-unit is used to obtain the first weight corresponding to the average similarity score, the second weight corresponding to the influence score, and the third weight corresponding to the information increment score.

[0073] The seventh determining subunit is used to determine the sample quality assessment score corresponding to the initial sample data based on the first weight, the second weight, the third weight, the average similarity score, the influence score, and the information increment score.

[0074] In one possible implementation, the above-mentioned filtering module includes:

[0075] The fifth determining unit is used to sort the multiple initial sample data according to the multiple sample quality assessment scores to obtain the sorting result;

[0076] The sixth determining unit is used to determine the first preset number of initial sample data with the highest sample quality evaluation scores in the above sorting results as multiple candidate sample data.

[0077] In one possible implementation, the first generation module described above includes:

[0078] The seventh determining unit is used to determine multiple second similarity scores between each of the above-mentioned unlabeled sample data and the above-mentioned multiple candidate sample data;

[0079] The eighth determining unit is used to determine multiple word coverage scores between each of the above-mentioned unlabeled sample data and the above-mentioned multiple candidate sample data;

[0080] The ninth determining unit is used to determine multiple corrected similarity scores corresponding to the multiple candidate sample data based on the multiple second similarity scores and the multiple word coverage scores.

[0081] The tenth determining unit is used to determine the target sample data from the multiple candidate sample data based on the multiple corrected similarity scores.

[0082] The eleventh determining unit is used to determine the real label corresponding to the above target sample data as the target label corresponding to the above sample data to be labeled.

[0083] In one possible implementation, the eighth determining unit mentioned above includes:

[0084] The first extraction subunit is used to extract the first vocabulary set included in the above-mentioned sample data to be labeled by a preset part-of-speech recognition tool;

[0085] The second extraction subunit is used to extract the second vocabulary set included in each candidate sample data from the above-mentioned multiple candidate sample data using the above-mentioned preset part-of-speech recognition tool.

[0086] The twelfth determining unit is used to determine multiple word coverage scores between the above-mentioned unlabeled sample data and the above-mentioned multiple candidate sample data based on the above-mentioned first word set and the second word set included in the above-mentioned candidate sample data.

[0087] In one possible implementation, the above-mentioned device further includes:

[0088] The second generation module is used to generate prompt information corresponding to the above-mentioned sample data to be labeled based on the target labels corresponding to each of the above-mentioned sample data to be labeled.

[0089] The training module is used to train the preset large language model based on the above-mentioned unlabeled sample data and the above-mentioned prompt information.

[0090] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory;

[0091] The aforementioned memory stores a computer program adapted to be loaded by the aforementioned processor and execute the steps of the method provided by the first aspect of the embodiments of this application or any possible implementation thereof.

[0092] Fourthly, embodiments of this application provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the steps of the method provided by the first aspect of the embodiments of this application or any possible implementation thereof.

[0093] This application embodiment acquires a labeled sample dataset and an unlabeled sample dataset. The labeled sample dataset includes multiple initial sample data and the true labels of each initial sample data. The unlabeled sample dataset includes multiple unlabeled sample data. Multiple sample quality assessment scores are determined for each of the initial sample data. Based on these scores, the initial sample data is filtered to obtain multiple candidate sample data. For each unlabeled sample data, a target label is generated based on the unlabeled sample data and the candidate sample data. By using the labeled initial sample data to filter out high-quality candidate sample data, the model references more effective samples when generating target labels. This incorporates feature information from candidate sample data during the label generation process for unlabeled sample data, making the generated target labels closer to the true labels, reducing the mislabeling rate, and improving label generation efficiency and accuracy. This, in turn, helps improve training efficiency during large language model training. Attached Figure Description

[0094] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0095] Figure 1 A schematic diagram of the structure of a label generation system provided in an exemplary embodiment of this application;

[0096] Figure 2 A flowchart illustrating a label generation method provided for an exemplary embodiment of this application;

[0097] Figure 3 A flowchart illustrating a method for determining multiple candidate sample data provided in an exemplary embodiment of this application;

[0098] Figure 4 A flowchart illustrating a target label determination method provided for an exemplary embodiment of this application;

[0099] Figure 5 A schematic diagram of a label generation apparatus provided for an exemplary embodiment of this application;

[0100] Figure 6 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this application. Detailed Implementation

[0101] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0102] The terms "first," "second," "third," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0103] Please refer to the following. Figure 1 The example illustrates a schematic diagram of the structure of a label generation system provided in an embodiment of this application. Figure 1 As shown, the system includes a terminal device 110 and a server 120, which are connected via a network, such as a wired or wireless network.

[0104] The terminal device 110 can be used to display a graphical user interface (GUI) for interacting with the user, such as downloading, installing, and running a corresponding client. In this embodiment, the terminal device 110 can also allow relevant personnel to upload labeled and unlabeled sample datasets. The terminal device 110 can send the labeled and unlabeled sample datasets to the server 120, allowing the server 120 to determine multiple sample quality assessment scores corresponding to the initial sample data. Based on these scores, the server 120 filters the initial sample data to obtain multiple candidate sample data. Then, for each unlabeled sample data, the server 120 generates a target label corresponding to each unlabeled sample data based on the unlabeled sample data and the multiple candidate sample data, and feeds the resulting target label back to the terminal device 110.

[0105] Optionally, the terminal device 110 can also directly determine multiple sample quality assessment scores corresponding to the multiple initial sample data based on the labeled sample dataset and unlabeled sample dataset uploaded by relevant personnel; based on the multiple sample quality assessment scores, the multiple initial sample data are filtered to obtain multiple candidate sample data; for each unlabeled sample data in the multiple unlabeled sample data, the target label corresponding to each unlabeled sample data is generated according to the multiple unlabeled sample data and the multiple candidate sample data.

[0106] Optionally, in the aforementioned labeled sample dataset, each initial sample data can correspond to one or more real labels. These real labels can be manually labeled data or data generated by automated or semi-automated labeling systems.

[0107] Optionally, the labeled sample dataset mentioned above is a dataset containing a relatively small amount of data, while the unlabeled sample dataset mentioned above is a dataset containing a relatively large amount of data. The labeled sample dataset is not a subset of the unlabeled sample dataset, and the labeled sample dataset mentioned above has all the characteristics of the unlabeled sample dataset, that is, the labeled sample data can represent the unlabeled sample dataset.

[0108] Optionally, the sample quality assessment score corresponding to the initial sample data is used to represent the quality or importance of the initial sample data. This sample quality assessment score can be calculated by combining multiple indicators, which can reflect the value and information contribution of the initial sample data to model training.

[0109] Optionally, the aforementioned candidate sample data are included in the aforementioned initial sample data, that is, the aforementioned candidate sample data are a subset obtained from the aforementioned labeled sample dataset.

[0110] Optionally, the target label corresponding to each unlabeled sample data is a predicted label generated for each unlabeled sample data. The target label represents the category to which the corresponding unlabeled sample data belongs or the label that should be assigned.

[0111] An exemplary embodiment of this application provides a tag generation method. This tag generation method can be applied to the aforementioned terminal device. Please refer to [reference needed] for details. Figure 2 The example illustrates a flowchart of a label generation method provided in an embodiment of this application. Figure 2 As shown, the label generation method includes the following steps S21-S24:

[0112] S21. Obtain labeled sample dataset and unlabeled sample dataset. The labeled sample dataset includes multiple initial sample data and the real labels of each initial sample data. The unlabeled sample dataset includes multiple sample data to be labeled.

[0113] In some embodiments, the initial sample data mentioned above can be statements, and the actual labels of the initial sample data can represent the category or attributes of the statement, etc.

[0114] In some embodiments, each initial sample data in the aforementioned labeled sample dataset may correspond to one or more real labels. These real labels may be manually labeled data or data generated by an automated or semi-automated labeling system.

[0115] In some embodiments, the labeled sample dataset is a dataset containing a small amount of data, and the unlabeled sample dataset is a dataset containing a large amount of data. The labeled sample dataset is not a subset of the unlabeled sample dataset, and the labeled sample dataset has all the characteristics of the unlabeled sample dataset, that is, the labeled sample data can represent the unlabeled sample dataset.

[0116] S22. Determine the sample quality assessment scores corresponding to the above initial sample data.

[0117] In some embodiments, the initial sample data and the sample quality assessment scores described above correspond one-to-one.

[0118] The sample quality assessment score corresponding to the aforementioned initial sample data is used to represent the quality or importance of the initial sample data. This sample quality assessment score can be calculated by combining multiple indicators, which can reflect the value and information contribution of the initial sample data to model training.

[0119] Optionally, the sample quality assessment score can be calculated by combining at least one of the following indicators: the average similarity score between the initial sample data and the multiple initial sample data, the influence score of the initial sample data in the labeled sample dataset, and the information increment score corresponding to the initial sample data, etc.

[0120] S23. Based on the above multiple sample quality assessment scores, the above multiple initial sample data are screened to obtain multiple candidate sample data.

[0121] The aforementioned candidate sample data are contained within the aforementioned initial sample data; that is, the candidate sample data are subsets selected from the aforementioned labeled sample dataset.

[0122] S24. For each unlabeled sample data in the above multiple unlabeled sample data, generate the target label corresponding to each unlabeled sample data based on the above multiple unlabeled sample data and the above multiple candidate sample data.

[0123] In this context, the target label for each unlabeled sample data is a predicted label generated for each unlabeled sample data. This target label represents the category to which the corresponding unlabeled sample data belongs or the label that should be assigned. Specifically, the target label is the result inferred by a model or algorithm based on the relationship between the unlabeled sample data and the candidate sample data (high-quality sample data with known true labels).

[0124] In S24, the process of generating target labels is essentially the automatic labeling of unlabeled sample data, assigning each sample a corresponding label, thus transforming it from unlabeled data into labeled data. These target labels can be used for subsequent large language model training, data analysis, or other applications.

[0125] This application embodiment acquires a labeled sample dataset and an unlabeled sample dataset. The labeled sample dataset includes multiple initial sample data and the true labels of each initial sample data. The unlabeled sample dataset includes multiple unlabeled sample data. Multiple sample quality assessment scores are determined for each of the initial sample data. Based on these scores, the initial sample data is filtered to obtain multiple candidate sample data. For each unlabeled sample data, a target label is generated based on the unlabeled sample data and the candidate sample data. By using the labeled initial sample data to filter out high-quality candidate sample data, the model references more effective samples when generating target labels. This incorporates feature information from candidate sample data during the label generation process for unlabeled sample data, making the generated target labels closer to the true labels, reducing the mislabeling rate, and improving label generation efficiency and accuracy. This, in turn, helps improve training efficiency during large language model training.

[0126] In some embodiments, in S22, determining the plurality of sample quality assessment scores corresponding to the plurality of initial sample data includes S221-S224:

[0127] S221. For each initial sample data in the above-mentioned multiple initial sample data, determine the average similarity score between the above-mentioned initial sample data and the above-mentioned multiple initial sample data.

[0128] The average similarity score between the initial sample data and the multiple initial sample data is used to measure the similarity between the initial sample data and the labeled sample dataset.

[0129] S222. Determine the influence score of the initial sample data in the labeled sample dataset.

[0130] The influence score of the initial sample data in the labeled sample dataset can be used to measure the contribution or influence of the initial sample data on model training and prediction.

[0131] S223. Determine the information increment score of the initial sample data in the labeled sample dataset.

[0132] The information increment score represents the amount of additional information provided by the initial sample data relative to other initial sample data. A higher information increment score indicates a greater difference in the predicted distribution of the current initial sample data compared to other initial sample data, providing more new information and making it more valuable for model training.

[0133] S224. Based on the above average similarity score, the above influence score and the above information increment score, determine the sample quality assessment score corresponding to the above initial sample data, and then obtain multiple sample quality assessment scores corresponding to the above multiple initial sample data.

[0134] In this embodiment of the application, by calculating the average similarity score, influence score and information increment score of the samples, the sample data that is valuable for model training can be effectively screened out, thereby reducing the interference of low-quality or duplicate information samples and improving the training effect and prediction accuracy of the model.

[0135] In some embodiments, in S221, for each initial sample data in the plurality of initial sample data, the average similarity score between the initial sample data and the plurality of initial sample data is determined, including S2211-S2213:

[0136] S2211. Determine the semantic vector corresponding to the above initial sample data.

[0137] In some embodiments, the text content of the initial sample data can be converted into semantic vectors using a relevant semantic embedding model (such as the sentence embedding model SBERT). Semantic vectors can effectively capture the features and semantic information of the initial sample data.

[0138] Alternatively, the semantic vectors corresponding to the initial sample data can be determined using all-distil Multi-Language Mini Language Model (all-MiniLM), such as all-MiniLM-L6-v2 (layer 6, version 2) and all-MiniLM-L12-v2 (layer 12, version 2); or using the All-Distilled Robustly Optimized BERT ApproachVersion 1 (all-distilroberta-v1) or the All-Masked and Permuted Network Base Version 2 (allmpnet-base-v2).

[0139] Optionally, a labeled sample dataset can be represented as D l The initial sample data and its true label in the labeled sample dataset can be represented as D. i The labeled sample dataset includes N initial sample data points, where i is any positive integer not less than N, representing the index of the initial sample data. Specifically, each initial sample data point and its true label D... i It can be represented as D i =(x i ,y i ), where x i Represents the initial sample data, y i Represents the initial sample data x i The true label.

[0140] The semantic vector e corresponding to each initial sample data is determined through the relevant semantic embedding model. i This leads to the semantic vector set E = {e1, ..., e} corresponding to the entire labeled sample dataset. N}, where e1~e N This represents the semantic vector corresponding to each initial sample data.

[0141] S2212. Based on the semantic vectors corresponding to the initial sample data, generate multiple first similarity scores between the initial sample data and the multiple initial sample data.

[0142] In some embodiments, the multiple first similarity scores are the similarity scores between the current initial sample data and all initial sample data in the labeled sample dataset (including the current initial sample data itself).

[0143] In some embodiments, multiple first similarity scores between the initial sample data and the multiple initial sample data can be generated based on the semantic vector corresponding to the initial sample data according to the cosine similarity formula.

[0144] In other embodiments, multiple first similarity scores between the aforementioned initial sample data and the aforementioned multiple initial sample data can also be calculated using Euclidean distance, Manhattan distance, Minkowski distance, Jaccard similarity, Pearson correlation coefficient, etc.

[0145] Optionally, the current initial sample data x i Compared with the labeled sample dataset D mentioned above l Any initial sample data x from all initial sample data (including the current initial sample data itself) j The similarity score can be represented as cosine(e i ,e j ).

[0146] S2213. Determine the average similarity score corresponding to the initial sample data based on the multiple first similarity scores mentioned above.

[0147] In some embodiments, in S2213, determining the average similarity score corresponding to the initial sample data based on the plurality of first similarity scores includes: taking the average of the plurality of first similarity scores as the average similarity score corresponding to the initial sample data.

[0148] Optionally, the average similarity score corresponding to the above initial sample data It can be represented as:

[0149]

[0150] The lower the average similarity score of the initial sample data, the more it reflects the difference between the initial sample data and other initial sample data in the entire labeled sample dataset.

[0151] In this embodiment, semantic vectors can extract deep semantic features of text data. During the processing, they can go beyond the similarity of surface words and make deeper semantic comparisons to more accurately measure the similarity between sample data and improve the quality of sample data screening.

[0152] In some embodiments, in S222, the influence score of the initial sample data in the labeled sample dataset is determined, including S2221-S2224:

[0153] S2221. Predict the labels of the labeled sample dataset by using a pre-set large language model, and obtain the first label prediction probability of the true labels of all initial sample data in the labeled sample dataset output by the large language model.

[0154] Among them, the aforementioned pre-defined large language model is a large language model that needs to be trained or optimized.

[0155] In some embodiments, the initial sample data of the labeled sample dataset corresponds one-to-one with the predicted probability of the first label.

[0156] In some embodiments, the labeled sample dataset (containing the current initial sample data) can be input into a preset large language model to obtain the output of the large language model. Then, prediction information for each initial sample data can be extracted from this output, and the predicted probability of the true label for each initial sample given by the large language model is denoted as the first label prediction probability. The labeled sample dataset D output by the large language model... l Any initial sample data x j Real Labels j The first label prediction probability can be expressed as P LM (y j |x j D i ), where y i This is any initial sample data from the labeled sample dataset.

[0157] S2222. Remove the current initial sample data from the labeled sample dataset to obtain a subset of labeled sample data.

[0158] The aforementioned subset of labeled sample data is a subset of the aforementioned dataset of labeled samples.

[0159] S2223. Using the above-mentioned large language model, the labels of the above-mentioned labeled sample data subset are predicted to obtain the second label prediction probability of the true labels of all initial sample data in the above-mentioned labeled sample data subset output by the above-mentioned large language model.

[0160] In some embodiments, the initial sample data in the labeled sample data subset above corresponds one-to-one with the predicted probability of the second label above.

[0161] In some embodiments, the aforementioned subset of labeled sample data (excluding the current initial sample data) can be input into a preset large language model to obtain the output of the large language model. Then, the prediction information of the initial sample data can be extracted from the output, and the prediction probability of the true label of each initial sample given by the large language model can be denoted as the second label prediction probability.

[0162] The large language model outputs any initial sample data x from the subset of labeled sample data. j Real Labels j The second label prediction probability can be expressed as P LM (y j |x j ), where y j It is any initial sample data in the subset of labeled sample data.

[0163] S2224. Determine the influence score of the initial sample data in the labeled sample dataset based on the first label prediction probability and the second label prediction probability.

[0164] In some embodiments, in S2224, determining the influence score of the initial sample data in the labeled sample dataset based on the first label prediction probability and the second label prediction probability includes: determining the difference between the first label prediction probability and the second label prediction probability as the influence score of the initial sample data in the labeled sample dataset.

[0165] Optionally, the influence score of the initial sample data in the labeled sample dataset can be expressed as:

[0166]

[0167] The higher the influence score of the initial sample data in the labeled sample dataset mentioned above, the greater the contribution of the initial sample to the final prediction output of the large language model.

[0168] In this embodiment, by comparing the model's prediction results with and without the current initial sample data, the initial sample data that has a significant impact on the model's prediction performance can be identified. This initial sample data is more representative or contains unique information, which helps the model learn classification rules better and improves model training efficiency.

[0169] In some embodiments, in S223, determining the information increment score of the initial sample data in the labeled sample dataset includes S2231-S2234:

[0170] S2231. Predict the labels of the labeled sample dataset by using a pre-set large language model, and obtain the first label prediction probability of the true labels of all initial sample data in the labeled sample dataset output by the large language model.

[0171] In some embodiments, the specific process of S2231 is consistent with the process of S2221 described above, and will not be repeated here.

[0172] S2232. Determine the entropy data of the initial sample data based on the predicted probability of the first label.

[0173] Among them, the entropy data of the initial sample data can be used to measure the uncertainty of the large language model's prediction of the current initial sample data.

[0174] In some embodiments, the entropy data H of the initial sample data described above can be determined using the following formula. θ (D i ):

[0175]

[0176] Where, p i In S2231, a pre-defined large language model is used to target the current initial sample data x. i Real Labels i The predicted probability of the first label.

[0177] For each initial sample data, the smaller the entropy data, the higher the certainty of the large language model's prediction for that initial sample data; the larger the entropy data, the lower the certainty of the large language model's prediction for that initial sample data.

[0178] S2233. Based on the predicted probability of the first label, determine the multiple cross-entropy data between the initial sample data and the multiple initial sample data.

[0179] In some embodiments, the initial sample data corresponds one-to-one with the multiple cross-entropy data. That is, the cross-entropy data between the current initial sample data and any initial sample data in the labeled sample dataset is calculated to obtain multiple cross-entropy data.

[0180] In some embodiments, the current initial sample data x can be calculated using the following formula. i Compared to any initial sample data x in the labeled sample dataset j Cross-entropy data H between θ (x j |D i ):

[0181]

[0182] Where, p j (x j In S2231, a pre-defined large language model is used to target the initial sample data x. j The first label prediction probability of the true label prediction. i (D iIn S2231, a pre-defined large language model is used to target the initial sample data x. i The first label prediction probability of the true label prediction.

[0183] S2234. Determine the information increment score corresponding to the initial sample data based on the above self-entropy data and the above multiple cross-entropy data.

[0184] In some embodiments, in S2234, determining the information increment score corresponding to the initial sample data based on the self-entropy data and the plurality of cross-entropy data includes:

[0185] Calculate the difference between each cross-entropy data point and its own entropy data in the above multiple cross-entropy data points, and then calculate the average of the differences between each cross-entropy data point and its own entropy data. This average value is determined as the information increment score corresponding to the current initial sample data.

[0186] Specifically, the initial sample data x i The corresponding information increment score info(D) i This can be represented as:

[0187]

[0188] In this embodiment, the information increment score reflects the additional information a given initial sample brings compared to other samples, helping the model identify which samples provide unique semantic information, thereby increasing the effectiveness of model training. Furthermore, by selecting samples with high information increment scores, the model's learning content can be enriched, enabling it to better cover diverse inputs and thus improve its generalization ability. Especially in scenarios with small sample sizes, the information increment score helps focus on high-value samples, improving the model's performance in situations of data scarcity.

[0189] In some embodiments, in S224, the determination of the sample quality assessment score corresponding to the initial sample data based on the average similarity score, the influence score, and the information increment score includes S2241-S2242:

[0190] S2241. Obtain the first weight corresponding to the average similarity score, the second weight corresponding to the influence score, and the third weight corresponding to the information increment score.

[0191] The first, second, and third weights are pre-set, and their specific values ​​can be flexibly set according to the actual situation. This application does not impose any specific restrictions on them.

[0192] S2242. Determine the sample quality assessment score corresponding to the initial sample data based on the first weight, the second weight, the third weight, the average similarity score, the influence score, and the information increment score.

[0193] In some embodiments, in S2242, determining the sample quality assessment score corresponding to the initial sample data based on the first weight, the second weight, the third weight, the average similarity score, the influence score, and the information increment score includes:

[0194] Calculate the first product of the first weight and the reciprocal of the average similarity score, calculate the second product of the second weight and the influence score, and calculate the third product of the third weight and the reciprocal of the information increment score; the sum of the first product, the second product, and the third product is determined as the sample quality assessment score corresponding to the initial sample data.

[0195] Optionally, the above initial sample data x j The corresponding sample quality assessment score (D) j This can be represented as:

[0196]

[0197] In this context, α1 is the first weight, α2 is the second weight, and α3 is the third weight.

[0198] In this embodiment, the three dimensions of sample evaluation are combined through linear transformation. By changing the weight coefficients of the dimensions, it can be adapted to different scenarios and sample concerns. At the same time, linear transformation has high interpretability, effectively improving the comprehensiveness, accuracy and reliability of sample data evaluation.

[0199] In some embodiments, in S23, the multiple initial sample data are screened based on the multiple sample quality assessment scores to obtain multiple candidate sample data, including S231-S232:

[0200] S231. Sort the initial sample data according to the sample quality assessment scores to obtain the sorting results.

[0201] In some embodiments, multiple initial sample data can be sorted according to the sample quality assessment scores from largest to smallest or from smallest to largest.

[0202] S232. The first preset number of initial sample data with the highest sample quality evaluation score in the above sorting results are determined as multiple candidate sample data.

[0203] In some embodiments, the first preset quantity m << N.

[0204] In some embodiments, the multiple candidate sample data constitute a candidate context example set D. m .

[0205] In this embodiment, by ranking the sample quality assessment scores, high-quality samples can be prioritized for model training, thereby improving the model's prediction accuracy and overall performance. The first preset quantity m is much smaller than the total number of samples N, indicating that only a small subset of high-quality samples are selected for training or use. This significantly reduces the amount of data while maintaining model performance, lowering training costs and computational resource consumption, allowing the model to complete training faster and thus improving training efficiency.

[0206] The above is the first stage of the label generation method provided in this application. Figure 3 A flowchart illustrating a method for determining multiple candidate sample data provided in an embodiment of this application is shown below. Figure 3 The method for determining these multiple candidate sample data includes the following S301-S308:

[0207] S301. Obtain labeled sample datasets and unlabeled sample datasets.

[0208] S302. Determine the average similarity score between the initial sample data and multiple initial sample data.

[0209] S303. Determine the influence score of the initial sample data in the labeled sample dataset.

[0210] S304. Determine the information increment score of the initial sample data in the labeled sample dataset.

[0211] S305. Obtain the first weight corresponding to the average similarity score, the second weight corresponding to the influence score, and the third weight corresponding to the information increment score.

[0212] S306. Determine the sample quality assessment score corresponding to the initial sample data based on the first weight, second weight, third weight, average similarity score, influence score, and information increment score, and then obtain multiple sample quality assessment scores.

[0213] S307. Sort multiple initial sample data according to multiple sample quality assessment scores to obtain the sorting results.

[0214] S308. The first preset number of initial sample data with the highest sample quality evaluation score in the sorting results are determined as multiple candidate sample data.

[0215] The specific steps of S301-S308 are the same as those of S21-S23 above, and will not be repeated here.

[0216] In this embodiment of the application, the first stage constructs a candidate context example set (including multiple candidate sample data) and filters out candidate sample data with higher sample quality evaluation scores, thereby reducing the amount of data processed in subsequent data processing and effectively saving computing resources.

[0217] Furthermore, the second stage of the label generation method provided in this application will be described below.

[0218] In some embodiments, in S24, the generation of target labels corresponding to the sample data to be labeled based on the sample data to be labeled and the plurality of candidate sample data includes S241-S245:

[0219] S241. Determine multiple second similarity scores between each of the above-mentioned unlabeled sample data and the above-mentioned multiple candidate sample data.

[0220] In some embodiments, the candidate sample data corresponds one-to-one with the second similarity score.

[0221] In some embodiments, the semantic vector corresponding to each unlabeled sample data and the semantic vector corresponding to each candidate sample data can be determined by the relevant semantic embedding model. Then, multiple second similarity scores between the unlabeled sample data and each candidate sample data can be calculated by the cosine similarity formula to obtain multiple second similarity scores between each unlabeled sample data and the above multiple candidate sample data.

[0222] S242. Determine the word coverage scores between each of the above-mentioned unlabeled sample data and the above-mentioned multiple candidate sample data.

[0223] The candidate sample data and the vocabulary coverage scores mentioned above correspond one-to-one. The vocabulary coverage score between the unlabeled sample data and each candidate sample data is used to measure the degree of overlap between the unlabeled sample data and each candidate sample data at the vocabulary level.

[0224] In some embodiments, in S242, the determination of multiple word coverage scores between the aforementioned unannotated sample data and the aforementioned multiple candidate sample data includes S2421-S2423:

[0225] S2421. Extract the first set of words included in each of the above-mentioned sample data to be labeled using a preset part-of-speech recognition tool.

[0226] In some embodiments, the first vocabulary set may include at least one vocabulary related to the shallow representation of the sample, such as nouns, adjectives, and adverbs, contained in the sample data to be labeled.

[0227] S2422. Extract the second vocabulary set included in each candidate sample data from the above-mentioned multiple candidate sample data using the above-mentioned preset part-of-speech recognition tool;

[0228] In some embodiments, the second vocabulary set may include at least one vocabulary related to the shallow representation of the sample, such as nouns, adjectives, and adverbs, contained in the candidate sample data.

[0229] S2423. Based on the first vocabulary set and the second vocabulary set included in each candidate sample data, determine the multiple vocabulary coverage scores between each unannotated sample data and the multiple candidate sample data.

[0230] Unlabeled sample datasets can be represented as D u The unlabeled sample data in the dataset can be represented as d i Candidate sample data can be represented as d j , j is the number of each candidate sample data, and j is a positive integer not less than the first preset number m.

[0231] In some embodiments, each sample data to be labeled d i With candidate sample data d j The vocabulary coverage score between the two is cover(d) i ,d j It can be determined by the following formula:

[0232]

[0233] Where ∈ is the offset coefficient, which serves to prevent the vocabulary coverage score from being 0 and to adjust the size ratio of the vocabulary coverage scores. Vocabulary coverage score cover(d) i ,d j The value range of ) is (∈, 1+∈], and the vocabulary coverage score can be used as a scaling factor for cosine similarity to avoid focusing only on the semantic information between samples when searching for similar samples. set(·) means extracting a vocabulary set using the above-mentioned preset part-of-speech recognition tool.

[0234] In this embodiment, by identifying shallow vocabulary (such as nouns, adjectives, adverbs, etc.) in the samples, the shallow features of the samples to be labeled can be understood more comprehensively, helping the model to focus on the diversity of samples during the labeling process. This vocabulary coverage can capture the representational features of different samples, improving the breadth and accuracy of sample selection. Furthermore, the vocabulary coverage score can be used as a scaling factor for cosine similarity, combining semantic information with the matching degree of shallow features when calculating sample similarity, avoiding focusing solely on deep semantic information. Therefore, the model can more comprehensively evaluate the similarity between samples, improving the accuracy of selecting candidate samples.

[0235] S243. Based on the above-mentioned multiple second similarity scores and multiple word coverage scores, determine multiple corrected similarity scores corresponding to the above-mentioned multiple candidate sample data.

[0236] In this system, each candidate sample data corresponds one-to-one with a corrected similarity score.

[0237] In some embodiments, in S243, determining multiple corrected similarity scores corresponding to the multiple candidate sample data based on the multiple second similarity scores and the multiple word coverage scores includes: taking the product of the second similarity score and the word coverage score between each unlabeled sample data and each candidate sample data as the corrected similarity score corresponding to the candidate sample data, so as to obtain multiple corrected similarity scores corresponding to the multiple candidate sample data.

[0238] In some downstream domain tasks, there are often specialized terms that are rarely found in the pre-training corpus of sentence vector models. By identifying the features of these specialized terms through a part-of-speech tagger and incorporating them into the similarity calculation, their weighting does not alter the contextual semantic information of the sentence itself. For example, the sample data "I love to eat tomatoes" and the sample data "I don't love to eat tomatoes" have a cosine similarity of less than 0, and their vocabulary coverage score is 1+∈. Multiplying them increases the absolute value of the cosine similarity, but the direction remains unchanged. The vocabulary coverage score aims to make semantically similar and lexically similar samples more similar, while reducing the similarity score for samples with dissimilar expressions but similar vocabulary, preventing them from being incorrectly selected into the context learning examples.

[0239] S244. Based on the above-mentioned multiple modified similarity scores, the target sample data is determined from the above-mentioned multiple candidate sample data.

[0240] In some embodiments, the second preset number (k) of candidate sample data with the highest corresponding corrected similarity scores among the above plurality of candidate sample data can be determined as target sample data, wherein the second preset number is less than the first preset number.

[0241] Specifically, the K-Nearest Neighbor (KNN) algorithm can be used to determine k target sample data from multiple candidate sample data.

[0242] S245. Determine the real label corresponding to the above target sample data as the target label corresponding to the above unlabeled sample data.

[0243] Adding the samples to be labeled and their target labels to the labeled sample dataset allows them to be used for subsequent model training or analysis.

[0244] In this embodiment, the similarity of semantic vectors and vocabulary coverage scores are used to accurately match the most similar candidate sample data for unlabeled sample data to be labeled, and the real labels are assigned to the sample to be labeled, thus realizing automatic labeling, saving manpower and time costs, effectively improving the efficiency of label generation, and this process makes full use of existing high-quality labeled sample data, thereby improving the accuracy of label generation.

[0245] The above describes the second stage of the label generation method provided in this application. Figure 4 This is a flowchart illustrating a target label determination method provided in an embodiment of this application, as shown below. Figure 4 The method for determining these multiple candidate sample data includes the following S401-S407:

[0246] S401. Determine multiple second similarity scores between each unlabeled sample data and multiple candidate sample data.

[0247] S402. Extract the first set of words included in each sample data to be labeled using a preset part-of-speech recognition tool.

[0248] S403. Extract the second vocabulary set included in each candidate sample data from multiple candidate sample data using a preset part-of-speech recognition tool.

[0249] S404. Based on the first vocabulary set and the second vocabulary set included in each candidate sample data, determine the multiple vocabulary coverage scores between each unannotated sample data and multiple candidate sample data.

[0250] S405. Determine multiple corrected similarity scores corresponding to multiple candidate sample data based on multiple second similarity scores and multiple word coverage scores.

[0251] S406. Determine the target sample data from multiple candidate sample data based on multiple corrected similarity scores.

[0252] S407. Determine the real label corresponding to the target sample data as the target label corresponding to the sample data to be labeled.

[0253] The specific steps of S401-S407 are the same as those of S24 above, and will not be repeated here.

[0254] In the second phase, this application focuses on building an example recaller, which recalls the k most similar samples to the currently input sample to be labeled from the candidate context example set as context examples. By combining the semantic relationship of the sample to be labeled with the syntactic information based on shallow surface words such as entities, patterns, and specific attributes, the corresponding target labels are automatically generated for the sample to be labeled, thereby improving the generalization of the large language model across multiple domains.

[0255] Furthermore, in some embodiments, the above method further includes:

[0256] Based on the target labels corresponding to the above-mentioned unlabeled sample data, generate prompt information corresponding to the above-mentioned unlabeled sample data; and train the preset large language model based on the above-mentioned unlabeled sample data and the above-mentioned prompt information.

[0257] In some embodiments, the sample data to be labeled and the corresponding target labels can be filled into a preset prompt template to generate specific prompt information. This creates prompt information containing the target label for each sample data to be labeled, which can guide the large language model to learn the correct output.

[0258] In this embodiment, target labels from the sample data to be labeled are used to generate corresponding prompts, and the large language model is trained based on these prompts, which can effectively improve the model's performance on specific tasks. This process makes full use of automatically generated labels, reduces the workload of manual labeling, and accelerates model iteration and application.

[0259] Please refer to the following. Figure 5 This is a schematic diagram of the structure of a label generation device provided in an exemplary embodiment of this application. Figure 5 As shown, the label generating apparatus 500 includes:

[0260] The acquisition module 501 is used to acquire a labeled sample dataset and an unlabeled sample dataset. The labeled sample dataset includes multiple initial sample data and the real labels of each initial sample data. The unlabeled sample dataset includes multiple sample data to be labeled.

[0261] The determination module 502 is used to determine the multiple sample quality assessment scores corresponding to the above multiple initial sample data;

[0262] The filtering module 503 is used to filter the multiple initial sample data based on the multiple sample quality assessment scores to obtain multiple candidate sample data;

[0263] The first generation module 504 is used to generate target labels corresponding to each of the multiple unlabeled sample data based on the multiple unlabeled sample data and the multiple candidate sample data.

[0264] In one possible implementation, the determining module 502 includes:

[0265] The first determining unit is used to determine the average similarity score between each initial sample data and the plurality of initial sample data for each initial sample data in the plurality of initial sample data;

[0266] The second determining unit is used to determine the influence score of the initial sample data in the labeled sample dataset.

[0267] The third determining unit is used to determine the information increment score of the initial sample data in the labeled sample dataset.

[0268] The fourth determining unit is used to determine the sample quality assessment score corresponding to the initial sample data based on the average similarity score, the influence score, and the information increment score, thereby obtaining multiple sample quality assessment scores corresponding to the multiple initial sample data.

[0269] In one possible implementation, the first determining unit includes:

[0270] The first determining subunit is used to determine the semantic vector corresponding to the aforementioned initial sample data;

[0271] A sub-unit is generated to generate multiple first similarity scores between the initial sample data and the multiple initial sample data based on the semantic vector corresponding to the initial sample data.

[0272] The second determining subunit is used to determine the average similarity score corresponding to the initial sample data based on the multiple first similarity scores.

[0273] In one possible implementation, the second determining unit includes:

[0274] The first prediction subunit is used to predict the labels of the labeled sample dataset by using a preset large language model, and to obtain the first label prediction probability of the true labels of all initial sample data in the labeled sample dataset output by the large language model.

[0275] The elimination sub-unit is used to remove the current initial sample data from the labeled sample dataset to obtain a subset of labeled sample data.

[0276] The second prediction subunit is used to predict the labels of the labeled sample data subset by using the large language model to obtain the second label prediction probability of the true labels of all initial sample data in the labeled sample data subset output by the large language model.

[0277] The third determining subunit is used to determine the influence score of the initial sample data in the labeled sample dataset based on the first label prediction probability and the second label prediction probability.

[0278] In one possible implementation, the third determining unit mentioned above includes:

[0279] The third prediction subunit is used to predict the labels of the labeled sample dataset by using a preset large language model, and to obtain the first label prediction probability of the true labels of all initial sample data in the labeled sample dataset output by the large language model.

[0280] The fourth determining subunit is used to determine the self-entropy data of the initial sample data based on the predicted probability of the first label.

[0281] The fifth determining subunit is used to determine multiple cross-entropy data between the initial sample data and the multiple initial sample data based on the predicted probability of the first label.

[0282] The sixth determining subunit is used to determine the information increment score corresponding to the initial sample data based on the self-entropy data and the multiple cross-entropy data.

[0283] In one possible implementation, the fourth determining unit mentioned above includes:

[0284] The sub-unit is used to obtain the first weight corresponding to the average similarity score, the second weight corresponding to the influence score, and the third weight corresponding to the information increment score.

[0285] The seventh determining subunit is used to determine the sample quality assessment score corresponding to the initial sample data based on the first weight, the second weight, the third weight, the average similarity score, the influence score, and the information increment score.

[0286] In one possible implementation, the filtering module 503 includes:

[0287] The fifth determining unit is used to sort the multiple initial sample data according to the multiple sample quality assessment scores to obtain the sorting result;

[0288] The sixth determining unit is used to determine the first preset number of initial sample data with the highest sample quality evaluation scores in the above sorting results as multiple candidate sample data.

[0289] In one possible implementation, the first generation module 504 described above includes:

[0290] The seventh determining unit is used to determine multiple second similarity scores between each of the above-mentioned unlabeled sample data and the above-mentioned multiple candidate sample data;

[0291] The eighth determining unit is used to determine multiple word coverage scores between each of the above-mentioned unlabeled sample data and the above-mentioned multiple candidate sample data;

[0292] The ninth determining unit is used to determine multiple corrected similarity scores corresponding to the multiple candidate sample data based on the multiple second similarity scores and the multiple word coverage scores.

[0293] The tenth determining unit is used to determine the target sample data from the multiple candidate sample data based on the multiple corrected similarity scores.

[0294] The eleventh determining unit is used to determine the real label corresponding to the above target sample data as the target label corresponding to the above sample data to be labeled.

[0295] In one possible implementation, the eighth determining unit mentioned above includes:

[0296] The first extraction subunit is used to extract the first vocabulary set included in the above-mentioned sample data to be labeled by a preset part-of-speech recognition tool;

[0297] The second extraction subunit is used to extract the second vocabulary set included in each candidate sample data from the above-mentioned multiple candidate sample data using the above-mentioned preset part-of-speech recognition tool.

[0298] The twelfth determining unit is used to determine multiple word coverage scores between the above-mentioned unlabeled sample data and the above-mentioned multiple candidate sample data based on the above-mentioned first word set and the second word set included in the above-mentioned candidate sample data.

[0299] In one possible implementation, the device 500 further includes:

[0300] The second generation module is used to generate prompt information corresponding to the above-mentioned sample data to be labeled based on the target labels corresponding to each of the above-mentioned sample data to be labeled.

[0301] The training module is used to train the preset large language model based on the above-mentioned unlabeled sample data and the above-mentioned prompt information.

[0302] The division of modules in the label generation device 500 described above is for illustrative purposes only. In other embodiments, the label generation device can be divided into different modules as needed to complete all or part of the functions of the label generation device described above. The implementation of each module in the label generation device provided in the embodiments of this specification can be in the form of a computer program. This computer program can run on a terminal or server. The program modules constituted by this computer program can be stored in the memory of the terminal or server. When the computer program is executed by a processor, it implements all or part of the steps of the label generation method described in the embodiments of this specification.

[0303] Please refer to the following. Figure 6 This is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this application. Figure 6As shown, the electronic device 600 may include a processor 610 and a memory 620, and may also include a user interface 630, a network interface 640 and a communication bus 650.

[0304] The processor 610 may include one or more processing cores. The processor 610 connects to various parts within the electronic device 600 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 620, and by calling data stored in the memory 620. Optionally, the processor 610 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 610 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system and applications; the GPU is responsible for rendering and drawing the content required for display on the screen; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 610 and may be implemented as a separate chip.

[0305] The memory 620 may include random access memory (RAM) or read-only memory. Optionally, the memory 620 may include a non-transitory computer-readable storage medium. The memory 620 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 620 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as receiving functions, control functions, etc.), instructions for implementing the various method embodiments described above, etc.; the data storage area may store data involved in the various method embodiments described above, etc. Optionally, the memory 620 may also be at least one storage device located remotely from the aforementioned processor 610. Figure 6 As shown, the memory 620, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and program instructions.

[0306] Optionally, the communication bus 650 is used to realize the connection and communication between these components. The user interface 630 may include a display screen, a camera, and may also include standard wired interfaces and wireless interfaces. The network interface 640 may optionally include standard wired interfaces and wireless interfaces (such as WIFI interfaces).

[0307] exist Figure 6 In the illustrated electronic device 600, the processor 610 can be used to call program instructions stored in the memory 620 and specifically perform the following operations:

[0308] Obtain labeled sample datasets and unlabeled sample datasets. The labeled sample dataset includes multiple initial sample data and the true labels of each initial sample data. The unlabeled sample dataset includes multiple sample data to be labeled.

[0309] Determine the sample quality assessment scores corresponding to the above initial sample data;

[0310] Based on the above multiple sample quality assessment scores, the above multiple initial sample data are screened to obtain multiple candidate sample data;

[0311] For each of the above unlabeled sample data, a target label corresponding to each of the above unlabeled sample data and the above candidate sample data is generated.

[0312] In one possible implementation, determining the multiple sample quality assessment scores corresponding to the multiple initial sample data includes:

[0313] For each of the aforementioned initial sample data, determine the average similarity score between the aforementioned initial sample data and the aforementioned multiple initial sample data.

[0314] Determine the influence score of the initial sample data in the labeled sample dataset.

[0315] Determine the information increment score of the initial sample data in the labeled sample dataset;

[0316] Based on the above average similarity score, the above influence score, and the above information increment score, the sample quality assessment score corresponding to the above initial sample data is determined, and then multiple sample quality assessment scores corresponding to the above multiple initial sample data are obtained.

[0317] In one possible implementation, determining the average similarity score between the initial sample data and the plurality of initial sample data includes:

[0318] Determine the semantic vector corresponding to the initial sample data mentioned above;

[0319] Based on the semantic vectors corresponding to the initial sample data, multiple first similarity scores are generated between the initial sample data and the multiple initial sample data.

[0320] The average similarity score corresponding to the initial sample data is determined based on the multiple first similarity scores mentioned above.

[0321] In one possible implementation, determining the influence score of the initial sample data in the labeled sample dataset includes:

[0322] By using a pre-defined large language model to predict the labels of the labeled sample dataset, the first label prediction probability of the true labels of all initial sample data in the labeled sample dataset output by the large language model is obtained.

[0323] The current initial sample data is removed from the labeled sample dataset to obtain a subset of labeled sample data;

[0324] By using the large language model described above to predict the labels of the labeled sample data subset, the second label prediction probability of the true labels of all initial sample data in the labeled sample data subset output by the large language model is obtained.

[0325] The influence score of the initial sample data in the labeled sample dataset is determined based on the predicted probabilities of the first label and the second label.

[0326] In one possible implementation, determining the information increment score of the initial sample data in the labeled sample dataset includes:

[0327] By using a pre-defined large language model to predict the labels of the labeled sample dataset, the first label prediction probability of the true labels of all initial sample data in the labeled sample dataset output by the large language model is obtained.

[0328] Based on the predicted probability of the first label, the entropy data of the initial sample data is determined.

[0329] Based on the predicted probability of the first label, determine multiple cross-entropy data between the initial sample data and the multiple initial sample data;

[0330] The information increment score corresponding to the initial sample data is determined based on the self-entropy data and the multiple cross-entropy data mentioned above.

[0331] In one possible implementation, determining the sample quality assessment score corresponding to the initial sample data based on the average similarity score, the influence score, and the information increment score includes:

[0332] Obtain the first weight corresponding to the average similarity score, the second weight corresponding to the influence score, and the third weight corresponding to the information increment score;

[0333] The sample quality assessment score corresponding to the initial sample data is determined based on the first weight, the second weight, the third weight, the average similarity score, the influence score, and the information increment score.

[0334] In one possible implementation, the aforementioned initial sample data and the aforementioned sample quality assessment scores correspond one-to-one. Based on the aforementioned multiple sample quality assessment scores, the aforementioned initial sample data are filtered to obtain multiple candidate sample data, including:

[0335] The initial sample data were sorted based on the quality assessment scores of the samples to obtain the sorting results;

[0336] The first preset number of initial sample data with the highest sample quality evaluation score in the above sorting results are determined as multiple candidate sample data.

[0337] In one possible implementation, the process of generating target labels corresponding to the unlabeled sample data based on the unlabeled sample data and the plurality of candidate sample data includes:

[0338] Determine multiple second similarity scores between each of the above-mentioned unlabeled sample data and the above-mentioned multiple candidate sample data;

[0339] Determine the word coverage scores between each of the above unannotated sample data and the above multiple candidate sample data;

[0340] Based on the aforementioned multiple second similarity scores and multiple word coverage scores, multiple corrected similarity scores are determined for the aforementioned multiple candidate sample data;

[0341] The target sample data is determined from the candidate sample data based on the above-mentioned multiple modified similarity scores;

[0342] The real labels corresponding to the target sample data are determined as the target labels corresponding to the sample data to be labeled.

[0343] In one possible implementation, determining the multiple word coverage scores between each of the unannotated sample data and the multiple candidate sample data includes:

[0344] The first set of words included in each of the above-mentioned sample data to be labeled is extracted using a preset part-of-speech recognition tool;

[0345] The second vocabulary set included in each candidate sample data in the above-mentioned multiple candidate sample data is extracted using the aforementioned preset part-of-speech recognition tool;

[0346] Based on the first vocabulary set and the second vocabulary set included in each candidate sample data, determine the vocabulary coverage scores between each unannotated sample data and the multiple candidate sample data.

[0347] In one possible implementation, the above method also includes:

[0348] Based on the target labels corresponding to the above sample data to be labeled, generate prompt information corresponding to the above sample data to be labeled.

[0349] The pre-set large language model is trained based on the above-mentioned unlabeled sample data and the above-mentioned prompt information.

[0350] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform one or more steps in the above embodiments. If the constituent modules of the above-described label generation apparatus are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium.

[0351] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).

[0352] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks. Unless otherwise specified, the technical features of this embodiment and its implementation can be combined arbitrarily.

[0353] The above-described embodiments are merely preferred embodiments of this application and are not intended to limit the scope of this application. Any modifications and improvements made by those skilled in the art to the technical solutions of this application without departing from the spirit of this application should fall within the protection scope defined by the claims of this application.

Claims

1. A label generation method, characterized in that, include: Obtain labeled sample datasets and unlabeled sample datasets. The labeled sample dataset includes multiple initial sample data and the real labels of each initial sample data. The unlabeled sample dataset includes multiple sample data to be labeled. The initial sample data is a text statement. The real labels of the initial sample data represent the category or attribute of the initial sample data. For each initial sample data in the plurality of initial sample data, determine the average similarity score between the initial sample data and the plurality of initial sample data; Determine the influence score of the initial sample data in the labeled sample dataset; The labels of the labeled sample dataset are predicted by a pre-set large language model, and the first label prediction probability of the true labels of all initial sample data in the labeled sample dataset is obtained by the output of the large language model. The entropy data of the initial sample data is determined based on the predicted probability of the first label. Based on the first label prediction probability, determine multiple cross-entropy data between the initial sample data and the plurality of initial sample data; Calculate the difference between each cross-entropy data and its own entropy data in the plurality of cross-entropy data, and calculate the average value of the difference between each cross-entropy data and its own entropy data, and determine the average value as the information increment score corresponding to the current initial sample data; The sample quality assessment score corresponding to the initial sample data is determined based on the average similarity score, the influence score, and the information increment score, thereby obtaining multiple sample quality assessment scores corresponding to the multiple initial sample data. Based on the multiple sample quality assessment scores, the multiple initial sample data are filtered to obtain multiple candidate sample data; For each unlabeled sample data in the plurality of unlabeled sample data, a target label corresponding to each unlabeled sample data is generated based on each unlabeled sample data and the plurality of candidate sample data.

2. The method as described in claim 1, characterized in that, Determining the average similarity score between the initial sample data and the plurality of initial sample data includes: Determine the semantic vector corresponding to the initial sample data; Based on the semantic vector corresponding to the initial sample data, generate multiple first similarity scores between the initial sample data and the plurality of initial sample data; The average similarity score corresponding to the initial sample data is determined based on the plurality of first similarity scores.

3. The method as described in claim 1, characterized in that, Determining the influence score of the initial sample data in the labeled sample dataset includes: The labels of the labeled sample dataset are predicted by a pre-set large language model, and the first label prediction probability of the true labels of all initial sample data in the labeled sample dataset is obtained by the output of the large language model. The current initial sample data is removed from the labeled sample dataset to obtain a subset of labeled sample data; The labels of the labeled sample data subset are predicted by the large language model to obtain the second label prediction probability of the true labels of all initial sample data in the labeled sample data subset output by the large language model. The influence score of the initial sample data in the labeled sample dataset is determined based on the first label prediction probability and the second label prediction probability.

4. The method as described in claim 1, characterized in that, The step of determining the sample quality assessment score corresponding to the initial sample data based on the average similarity score, the influence score, and the information increment score includes: Obtain the first weight corresponding to the average similarity score, the second weight corresponding to the influence score, and the third weight corresponding to the information increment score; The sample quality assessment score corresponding to the initial sample data is determined based on the first weight, the second weight, the third weight, the average similarity score, the influence score, and the information increment score.

5. The method as described in claim 1, characterized in that, The initial sample data and the sample quality assessment scores are in one-to-one correspondence. Based on the multiple sample quality assessment scores, the multiple initial sample data are filtered to obtain multiple candidate sample data, including: The initial sample data are sorted according to the multiple sample quality assessment scores to obtain the sorting results; The first preset number of initial sample data with the highest sample quality evaluation score in the sorting results are determined as multiple candidate sample data.

6. The method as described in claim 1, characterized in that, The step of generating target labels corresponding to each sample data to be labeled based on each sample data to be labeled and the plurality of candidate sample data includes: Determine multiple second similarity scores between each unlabeled sample data and the multiple candidate sample data; Determine multiple word coverage scores between each unannotated sample data and the multiple candidate sample data; Based on the plurality of second similarity scores and the plurality of word coverage scores, determine the plurality of corrected similarity scores corresponding to the plurality of candidate sample data; The target sample data is determined from the multiple candidate sample data based on the multiple modified similarity scores; The real label corresponding to the target sample data is determined as the target label corresponding to the sample data to be labeled.

7. The method as described in claim 6, characterized in that, Determining the word coverage scores between each unlabeled sample data and the multiple candidate sample data includes: The first set of words included in each sample data to be labeled is extracted using a preset part-of-speech recognition tool; The second vocabulary set included in each candidate sample data in the plurality of candidate sample data is extracted by the preset part-of-speech recognition tool; Based on the first vocabulary set and the second vocabulary set included in each candidate sample data, determine multiple vocabulary coverage scores between each unannotated sample data and the multiple candidate sample data.

8. The method as described in claim 1, characterized in that, The method further includes: Based on the target labels corresponding to each sample data to be labeled, generate prompt information corresponding to the sample data to be labeled; The preset large language model is trained based on the sample data to be labeled and the prompt information.

9. A label generating apparatus, characterized in that, include: The acquisition module is used to acquire labeled sample datasets and unlabeled sample datasets. The labeled sample datasets include multiple initial sample data and the real labels of each initial sample data. The unlabeled sample datasets include multiple sample data to be labeled. The initial sample data are text statements. The real labels of the initial sample data represent the category or attribute of the initial sample data. The determination module is used to determine multiple sample quality assessment scores corresponding to the multiple initial sample data; The filtering module is used to filter the multiple initial sample data based on the multiple sample quality assessment scores to obtain multiple candidate sample data; The first generation module is used to generate a target label corresponding to each unlabeled sample data based on each unlabeled sample data and the multiple candidate sample data for each unlabeled sample data in the plurality of unlabeled sample data; The determining module includes: The first determining unit is configured to determine the average similarity score between each initial sample data and the plurality of initial sample data for each initial sample data in the plurality of initial sample data; The second determining unit is used to determine the influence score of the initial sample data in the labeled sample dataset; The third determining unit is used to determine the information increment score of the initial sample data in the labeled sample dataset; The fourth determining unit is used to determine the sample quality assessment score corresponding to the initial sample data based on the average similarity score, the influence score and the information increment score, thereby obtaining multiple sample quality assessment scores corresponding to the multiple initial sample data. The third determining unit includes: The third prediction subunit is used to predict the labels of the labeled sample dataset using a preset large language model, and to obtain the first label prediction probability of the true labels of all initial sample data in the labeled sample dataset output by the large language model. The fourth determining subunit is used to determine the entropy data of the initial sample data based on the predicted probability of the first label; The fifth determining subunit is used to determine multiple cross-entropy data between the initial sample data and the plurality of initial sample data based on the first label prediction probability; The sixth determining subunit is used to calculate the difference between each cross-entropy data and its own entropy data in the plurality of cross-entropy data, and to calculate the average value of the difference between each cross-entropy data and its own entropy data, and to determine the average value as the information increment score corresponding to the current initial sample data.

10. An electronic device, characterized in that, include: Processor and memory; The memory stores a computer program adapted to be loaded by the processor and to execute the steps of the method as claimed in any one of claims 1 to 8.

11. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the steps of the method as described in any one of claims 1 to 8.

Images