A data annotation method, system, terminal, and medium based on a large model

Abstract

This invention relates to the field of data annotation, specifically disclosing a data annotation method, system, terminal, and medium based on a large model, comprising: acquiring a standardized dataset; loading a fine-tuned domain model to perform batch pre-annotation on the standardized dataset, obtaining pre-annotation results for the standardized dataset; calculating the prediction uncertainty of each sample in the standardized dataset, and selecting several samples based on the prediction uncertainty to form a first dataset to be reviewed; predicting the contribution of each sample in the standardized dataset to the improvement of the domain model, and selecting several samples based on the improvement contribution to form a second dataset to be reviewed; taking the union of the first and second datasets to be reviewed to generate a target dataset to be reviewed, and manually reviewing the pre-annotation results of each sample in the target dataset to be reviewed; and obtaining the annotation results of the standardized dataset based on the manual review results. This invention improves the efficiency and accuracy of data annotation.

Description

Technical Field

[0001] This invention relates to the field of data annotation, specifically to a data annotation method, system, terminal, and medium based on a large model. Background Technology

[0002] With the rapid development of artificial intelligence technology, high-quality datasets are the core foundation for training and optimizing models, and the accuracy and efficiency of dataset annotation directly affect model performance. Traditional dataset annotation methods mainly rely on manual annotation, which is not only time-consuming and labor-intensive with extremely high human costs, but also inefficient, making it difficult to meet the needs of massive data annotation. With the widespread application of deep learning models, model-assisted automatic annotation methods have gradually emerged. However, existing automatic annotation technologies have many problems, such as insufficient annotation accuracy, difficulty in adapting to the characteristics of data in different fields, and a lack of comprehensive consideration of the reliability of annotation results and the value of model optimization, resulting in inconsistent annotation quality and failing to fully leverage the role of labeled data in improving model performance. Therefore, there is an urgent need for a data annotation method that is efficient, accurate, and can also meet the needs of model optimization. Summary of the Invention

[0003] To address the aforementioned issues, this invention provides a data annotation method, system, terminal, and medium based on a large model, thereby improving data annotation efficiency and accuracy.

[0004] In a first aspect, the technical solution of the present invention provides a data annotation method based on a large model, comprising the following steps:

[0005] Obtain the dataset to be labeled, and preprocess the dataset to be labeled to generate a standardized dataset;

[0006] The fine-tuned domain model is loaded and pre-labeled in batches on the standardized dataset to obtain the pre-labeling results of the standardized dataset;

[0007] The pre-labeled results are analyzed, the prediction uncertainty of each sample in the standardized dataset is calculated, and a number of samples are selected based on the prediction uncertainty to form the first dataset to be reviewed.

[0008] Predict the contribution of each sample in the standardized dataset to the improvement of the domain model, and select several samples based on the improvement contribution to form a second dataset to be reviewed.

[0009] The first dataset to be reviewed and the second dataset to be reviewed are combined to generate the target dataset to be reviewed. The pre-labeled results of each sample in the target dataset to be reviewed are then manually reviewed.

[0010] The annotation results of the standardized dataset are obtained based on the results of manual review.

[0011] In one optional implementation, the pre-labeled results are analyzed to calculate the prediction uncertainty for each sample in the standardized dataset, specifically including:

[0012] Obtain the class probability output of the sample;

[0013] The entropy value of a sample is calculated based on the category probability output using the entropy method.

[0014] The confidence uncertainty score of a sample is calculated based on the category probability output using the lowest confidence method.

[0015] The marginal uncertainty fraction of a sample is calculated based on the class probability output using the marginal sampling method.

[0016] In one optional implementation, a number of samples are selected to form a first dataset to be reviewed based on the prediction uncertainty, specifically including:

[0017] The sample's prediction uncertainty assessment value is obtained by weighting and summing the entropy value, confidence uncertainty score, and marginal uncertainty score.

[0018] The samples in the standardized dataset are sorted in descending order of their prediction uncertainty assessment values ​​to generate the first sample sequence.

[0019] Calculate the mean and standard deviation of the prediction uncertainty assessment values ​​in the first sample sequence;

[0020] The first dynamic threshold is determined based on the mean and standard deviation.

[0021] The first candidate set consists of samples whose prediction uncertainty assessment value is greater than or equal to the first dynamic threshold;

[0022] Extract feature vectors from the samples in the first candidate set, and then use a fast clustering algorithm to cluster the samples based on the feature vectors;

[0023] In each cluster, at least one sample with the highest prediction uncertainty assessment value is selected as the sample to be reviewed, and all the selected samples constitute the first dataset to be reviewed.

[0024] In one optional implementation, predicting the contribution of each sample in the standardized dataset to the improvement of the domain model specifically includes:

[0025] Calculate the gradient of the domain model loss function with respect to the domain model parameters for the sample, and calculate the L2 norm of the gradient as the first initial contribution score;

[0026] Calculate the expected changes in the domain model parameters for single-step gradient descent for the sample, and calculate the norm of these parameter changes as the second initial contribution score.

[0027] The density of the sample is calculated using kernel density estimation, and the first initial contribution score and the second initial contribution score are adjusted using this density to obtain the first contribution score and the second contribution score.

[0028] In one optional implementation, a number of samples are selected based on their contribution to the improvement to form a second dataset to be reviewed, specifically including:

[0029] The contribution evaluation value of the sample is obtained by weighted summing of the first contribution score and the second contribution score;

[0030] The samples in the standardized dataset are sorted in descending order of contribution evaluation value to generate a second sample sequence;

[0031] Calculate the mean and standard deviation of the contribution assessment values ​​in the second sample sequence;

[0032] The second dynamic threshold is determined based on the mean and standard deviation.

[0033] Samples with contribution evaluation values ​​greater than or equal to the second dynamic threshold constitute the second candidate set;

[0034] Extract feature vectors from the samples in the second candidate set, and then use a fast clustering algorithm to cluster the samples based on the feature vectors;

[0035] In each cluster, at least one sample with the highest contribution evaluation value is selected as the sample to be reviewed, and all the selected samples constitute the second dataset to be reviewed.

[0036] In one optional implementation, the annotation results of the standardized dataset are obtained based on the results of manual review, specifically including:

[0037] Check whether the number of manually corrected annotations exceeds a threshold;

[0038] If not, update the manually corrected results in the pre-annotation results to the manually corrected annotation results to obtain the final annotation results;

[0039] If so, the corrected annotation results and corresponding samples are used to form a training set, and the domain model is retrained using this training set.

[0040] The pre-labeled results of the standardized dataset are obtained again using the retrained domain model until the number of manually corrected labels is less than a threshold. Then, the manually corrected results in the current pre-labeled results are updated to the current manually corrected labels to obtain the final labeling results.

[0041] In one optional implementation, the preprocessing of the dataset to be labeled includes:

[0042] Detect the format of the data to be labeled;

[0043] If the data is in text format, then the data to be labeled will be processed by word segmentation and stop word removal.

[0044] If the data is in image format, it will be normalized and resized.

[0045] If the data is in audio format, noise reduction and segmentation are performed on the data to be labeled.

[0046] Secondly, the technical solution of the present invention provides a data annotation system based on a large model, comprising:

[0047] The dataset acquisition module is used to acquire the dataset to be labeled and to preprocess the dataset to be labeled to generate a standardized dataset.

[0048] The pre-labeling processing module is used to load the fine-tuned domain model to perform batch pre-labeling on the standardized dataset and obtain the pre-labeling results of the standardized dataset.

[0049] The audit data generation module is used to analyze the pre-labeled results, calculate the prediction uncertainty of each sample in the standardized dataset, select several samples based on the prediction uncertainty to form the first audit dataset, predict the contribution of each sample in the standardized dataset to the improvement of the domain model, and select several samples based on the improvement contribution to form the second audit dataset.

[0050] The manual review module is used to take the union of the first dataset to be reviewed and the second dataset to be reviewed to generate the target dataset to be reviewed, and to manually review the pre-labeled results of each sample in the target dataset to be reviewed.

[0051] The annotation result generation module is used to obtain annotation results for standardized datasets based on the results of manual review.

[0052] Thirdly, the technical solution of the present invention provides a terminal, comprising:

[0053] Memory, used to store data annotation programs based on large models;

[0054] A processor, configured to implement the steps of the large model-based data annotation method as described above when executing the large model-based data annotation program.

[0055] Fourthly, the present invention provides a computer-readable storage medium storing a data annotation program based on a large model, wherein the data annotation program based on the large model is executed by a processor to implement the steps of the data annotation method based on the large model as described in any of the above claims.

[0056] As can be seen from the above technical solution, this application has the following advantages: First, the data is pre-labeled using a fine-tuned domain model. Then, based on the prediction uncertainty of each sample and its contribution to the improvement of the domain model, several pre-labeled samples are selected as the results to be reviewed, which are then manually reviewed. Finally, the labeling results of the standardized dataset are obtained based on the results of the manual review. This invention improves labeling efficiency, reduces the workload of manual labeling, and lowers labeling costs by fine-tuning the domain model and using it for batch pre-labeling. Furthermore, in the pre-labeling result processing stage, samples are selected to form the first dataset to be reviewed by calculating the prediction uncertainty, accurately screening out samples with low reliability of the labeling results, ensuring that the manual review focuses on key data and improves labeling accuracy. The second dataset to be reviewed is selected based on the contribution of the samples to the improvement of the domain model, which can uncover samples with important value for model optimization, achieve the fit between labeled data and model training, and effectively improve model performance. By taking the union of the two datasets to be reviewed to generate the target dataset to be reviewed, targeted manual review is carried out, which not only ensures the labeling quality but also improves the review efficiency, ultimately obtaining high-quality labeling results, further improving labeling accuracy, and providing high-quality data support for subsequent model training. Attached Figure Description

[0057] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a schematic diagram of a data annotation method based on a large model provided in an embodiment of the present invention.

[0059] Figure 2 This is a schematic block diagram of a data annotation system based on a large model, provided as an embodiment of the present invention.

[0060] Figure 3 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present invention. Detailed Implementation

[0061] To make the purpose, features, and advantages of this application more apparent and understandable, specific embodiments and accompanying drawings will be used to clearly and completely describe the technical solution protected by this application. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0062] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs. The terms used in the description of the present invention in this application are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0063] Figure 1 FIG. is a schematic flowchart of a data annotation method based on a large model provided by an embodiment of the present invention. Among them, Figure 1 The execution subject can be a data annotation system based on a large model. The data annotation method based on a large model provided by an embodiment of the present invention is executed by a computer device. Correspondingly, the data annotation system based on a large model runs in the computer device. According to different requirements, the order of steps in this flowchart can be changed, and some can be omitted.

[0064] As Figure 1 shown, the method includes the following steps.

[0065] S1. Obtain the dataset to be annotated, and preprocess the dataset to be annotated to generate a standardized dataset.

[0066] Obtain the dataset to be annotated, and perform preprocessing operations such as word segmentation and stop word removal, normalized size adjustment, noise reduction and segmentation on data in different formats such as text, pictures, and audio to generate a standardized dataset.

[0067] Specifically, detect the format of the data to be annotated; if it is in text format, perform word segmentation and stop word removal on the data to be annotated; if it is in picture format, perform normalization and size adjustment on the data to be annotated; if it is in audio format, perform noise reduction and segmentation on the data to be annotated.

[0068] In some optional embodiments, multi-modal data to be annotated such as text, pictures, and audio are obtained through interface calls, file imports, etc., supporting the input of structured (such as tables) and unstructured data (such as documents, video frames).

[0069] For text data, a word segmentation tool is used to split the sentence into word vectors, and meaningless words such as "de" and "le" are filtered through a stop word list. At the same time, normalization processing such as case unification and special symbol cleaning is performed.

[0070] For picture data, the OpenCV library is used for pixel value normalization (scaled to the [0,1] interval), the size is adjusted according to the model input requirements, and grayscale conversion or channel normalization is performed.

[0071] For audio data, Fourier transform is used for noise reduction processing, the audio segment is segmented through the VAD (voice activity detection) algorithm, and feature vectors such as Mel frequency cepstral coefficients (MFCC) are extracted.

[0072] Unify data of different formats into an input format that the model can recognize, such as converting text into ID sequences, images into tensors, and audio into feature matrices, and storing them in standard data formats such as JSON or TFRecord.

[0073] S2 loads the fine-tuned domain model and performs batch pre-labeling on the standardized dataset to obtain the pre-labeling results of the standardized dataset.

[0074] The finely tuned domain model is loaded, and batch pre-labeling is performed on a standardized dataset to obtain pre-labeled results. Specifically, firstly, based on a general large model, supervised fine-tuning is performed using a small amount of labeled data within the domain, optimizing the objective function to cross-entropy loss or mean squared error. Then, for professional domains such as medicine and law, domain dictionaries and knowledge graphs are introduced to adjust parameter weights and improve the ability to recognize professional terms.

[0075] A batch processing mechanism is adopted, and GPU parallel computing is used to accelerate inference. The class probability distribution or regression value of each sample is generated through model forward propagation. When outputting pre-labeled results, the confidence score (such as softmax probability) of each predicted class is recorded to provide a data foundation for subsequent uncertainty analysis.

[0076] S3. Analyze the pre-labeled results, calculate the prediction uncertainty of each sample in the standardized dataset, and select several samples based on the prediction uncertainty to form the first dataset to be reviewed.

[0077] The pre-labeling results were analyzed, and the prediction uncertainty-related indicators of the samples were calculated using the entropy method, the lowest confidence method, and the marginal sampling method, respectively. These indicators were then weighted and summed to obtain the prediction uncertainty assessment value. After ranking, a dynamic threshold was determined based on the mean and standard deviation, and samples with high uncertainty were selected to form a candidate set. The candidate set samples were then clustered, and the samples with the highest prediction uncertainty in each cluster were selected to form the first dataset to be reviewed.

[0078] S4, predict the contribution of each sample in the standardized dataset to the improvement of the domain model, and select several samples based on the improvement contribution to form the second dataset to be reviewed.

[0079] The contribution of samples to the improvement of the domain model is calculated, including a first initial contribution score based on the L2 norm of the loss function gradient and a second initial contribution score based on the expected parameter change norm of single-step gradient descent. These two scores are then adjusted using kernel density. The adjusted contribution scores are weighted and summed to obtain a contribution evaluation value. This value is then sorted and combined with the mean and standard deviation to determine a dynamic threshold, selecting high-contribution samples to form a candidate set. The candidate set samples are clustered, and the samples with the highest contribution in each cluster are selected to form a second dataset awaiting review.

[0080] S5: Take the union of the first dataset to be reviewed and the second dataset to be reviewed to generate the target dataset to be reviewed. Then, manually review the pre-labeled results of each sample in the target dataset to be reviewed.

[0081] The union of the first and second datasets to be reviewed is taken, and duplicates are removed using hash values ​​to avoid reviewing the same data repeatedly. Samples can be grouped by type (text / image / audio) or domain label (e.g., lungs / heart in medical images) to facilitate division of labor among reviewers.

[0082] It provides a visual review interface that displays pre-labeling results, model confidence, uncertainty indicators, and contribution scores to assist reviewers in their judgment. It supports batch labeling, historical labeling, and marking of disputed samples. Review results are stored in real-time in JSON format, including original data, pre-labeling results, manually corrected values, and review timestamps.

[0083] S6, obtain the annotation results of the standardized dataset based on the results of manual review.

[0084] The number of manually corrected annotations is counted. If the correction rate (number of corrections / total number of reviews) is lower than a preset threshold (e.g., 10%), the erroneous items in the pre-annotated results are directly replaced with manually corrected values ​​to generate the final labeled dataset. If the correction rate exceeds the threshold, the corrected samples and their labels are combined into a new training set, and the domain model is retrained using transfer learning (freezing the base layer and fine-tuning the task layer). The S2-S6 process is repeated until the correction rate reaches the target.

[0085] In some optional implementations, step S3 involves analyzing the pre-labeled results and calculating the prediction uncertainty for each sample in the standardized dataset, specifically including the following steps.

[0086] S31.1, obtain the class probability output of the sample.

[0087] Obtaining domain model for samples Category probability output ,in Indicates the first Categories .

[0088] S31.2, calculate the entropy value of the sample based on the category probability output using the entropy method.

[0089] The entropy value of the probability distribution output by the model is calculated. The higher the entropy value, the greater the uncertainty.

[0090] The formula for calculating entropy is:

[0091]

[0092] In the formula, For the sample The uncertainty, i.e., the entropy value (in bits). Predicting samples for domain models Category The probability, This represents the total number of categories.

[0093] S31.3 calculates the confidence uncertainty score of a sample based on the class probability output using the lowest confidence method.

[0094] Select the sample with the lowest predicted probability by the model, such as the sample with the highest probability class confidence close to 0.5 in a classification task. Alternatively, directly take the complement of the model's highest predicted probability; the larger the value, the more uncertain the prediction.

[0095] Confidence Uncertainty Score The calculation formula is as follows:

[0096]

[0097] S31.4 calculates the marginal uncertainty score of a sample based on the class probability output using the marginal sampling method.

[0098] The difference between the probabilities of the two most likely categories predicted by the model is calculated. The smaller the difference, the higher the uncertainty.

[0099] Specifically, regarding probability distributions Sort by value in descending order to get the two highest probabilities. and .

[0100] Boundary Uncertainty Fraction The calculation formula is as follows:

[0101]

[0102] Furthermore, in step S3, several samples are selected to form the first dataset to be reviewed based on the prediction uncertainty, which specifically includes the following steps.

[0103] S32.1, the sample's entropy value, confidence uncertainty score, and marginal uncertainty score are weighted and summed to obtain the sample's prediction uncertainty assessment value.

[0104] S32.2 Sort the samples in the standardized dataset in descending order of the prediction uncertainty assessment value to generate the first sample sequence.

[0105] S32.3 Calculate the mean and standard deviation of the prediction uncertainty assessment values ​​in the first sample sequence.

[0106] S32.4, Determine the first dynamic threshold based on the mean and standard deviation.

[0107] Specifically, the formula for calculating the first dynamic threshold is as follows:

[0108]

[0109] In the formula, The first dynamic threshold, The mean calculated in step S32.3, The standard deviation calculated in step S32.3, This is the adjustment coefficient.

[0110] The first dynamic threshold is automatically adjusted according to the data distribution, avoiding the inefficiency caused by selecting a fixed ratio.

[0111] S32.5, the samples whose prediction uncertainty assessment value is greater than or equal to the first dynamic threshold constitute the first candidate set.

[0112] S32.6 Extract feature vectors from the samples in the first candidate set, and use a fast clustering algorithm to cluster the samples based on the feature vectors.

[0113] For example, BERT text embeddings and CNN image features are extracted from the first candidate set. Clustering ensures coverage of different data patterns and avoids repeatedly labeling similar samples.

[0114] S32.7 In each cluster, select at least one sample with the highest prediction uncertainty assessment value as the sample to be reviewed, and all the selected samples constitute the first dataset to be reviewed.

[0115] Select the highest-ranked sample from each cluster to ensure coverage of different data distribution patterns. If a cluster accounts for more than 30% of the samples, the second-highest-ranked sample from that cluster can be selected to prevent large clusters from dominating the data.

[0116] In some alternative implementations, step S4, which predicts the contribution of each sample in the standardized dataset to the improvement of the domain model, specifically includes the following steps.

[0117] S41.1 Calculate the gradient of the domain model loss function with respect to the domain model parameters for the sample, and calculate the L2 norm of the gradient as the first initial contribution score.

[0118] The larger the gradient of a sample with respect to the model's loss function, the more significant its impact on parameter updates and the higher its contribution.

[0119] Let the domain model parameters be The loss function is , among which is sample, It's a real label.

[0120] For the sample Calculate the gradient of the loss function with respect to the parameters:

[0121]

[0122] Calculate the gradient norm, and use the L2 norm of the gradient as the first initial contribution score. :

[0123]

[0124] In the formula, For the domain model parameter dimension, For the first The gradient components of each parameter.

[0125] S41.2 Calculate the expected changes in the domain model parameters for single-step gradient descent on the sample, and calculate the norm of these parameter changes as the second initial contribution score.

[0126] After predicting labeled samples, the magnitude of the model parameter update is considered; the larger the magnitude, the higher the contribution.

[0127] Assuming labeled samples The real label is Calculate the expected parameter changes in a single step of gradient descent:

[0128]

[0129] In the formula, This is the learning rate.

[0130] Calculate the change norm as the second initial contribution score. :

[0131]

[0132] S41.3, use kernel density estimation to calculate the sample density, and use this density to adjust the first initial contribution score and the second initial contribution score to obtain the first contribution score and the second contribution score.

[0133] Calculate the sample using kernel density estimation (KDE). Density:

[0134]

[0135] In the formula, For kernel function, For bandwidth parameters, This represents the number of samples.

[0136] First contribution score ;

[0137] Second contribution score In the formula To adjust the parameters.

[0138] Furthermore, in step S4, several samples are selected based on the improvement contribution to form a second dataset to be reviewed, which specifically includes the following steps.

[0139] S42.1, the first contribution score and the second contribution score of the sample are weighted and summed to obtain the contribution evaluation value of the sample.

[0140] S42.2 Sort the samples in the standardized dataset in descending order of contribution evaluation value to generate the second sample sequence.

[0141] S42.3 Calculate the mean and standard deviation of the contribution assessment values ​​in the second sample sequence.

[0142] S42.4, Determine the second dynamic threshold based on the mean and standard deviation.

[0143] S42.5, samples with contribution evaluation values ​​greater than or equal to the second dynamic threshold are formed into a second candidate set.

[0144] S42.6 Extract feature vectors from the samples in the second candidate set, and use a fast clustering algorithm to cluster the samples based on the feature vectors.

[0145] S42.7 In each cluster, select at least one sample with the highest contribution evaluation value as the sample to be reviewed, and all selected samples constitute the second dataset to be reviewed.

[0146] This process is similar to steps S32.1 to S32.7, and will not be described again here.

[0147] In some optional implementations, step S6 obtains the annotation results of the standardized dataset based on the results of manual review, specifically including the following steps.

[0148] S61, detect whether the number of manually corrected annotation results exceeds the threshold.

[0149] Specifically, a proportional threshold or an absolute threshold can be set based on the size of the dataset.

[0150] S62, if not, update the manually corrected results in the pre-annotation results to the manually corrected annotation results to obtain the final annotation results.

[0151] For uncorrected samples, the pre-labeled results are retained. For corrected samples, the labels are updated to manual labels, enabling direct output of results for small-scale corrections and shortening the labeling cycle. For large-scale corrections, the domain model needs to be retrained, as in step S63.

[0152] S63, if so, construct a training set from the corrected annotation results and corresponding samples, and retrain the domain model using this training set.

[0153] You can directly use the corrected samples for retraining, or you can add the corrected samples to the current training samples to form the total samples for retraining, thereby achieving enhanced training in areas with concentrated model errors and improving efficiency.

[0154] S64: Use the retrained domain model to obtain the pre-labeled results of the standardized dataset again until the number of manually corrected labels is less than the threshold. Then, update the manually corrected results in the current pre-labeled results to the current manually corrected labels to obtain the final labeling results.

[0155] Specifically, the loop terminates when the manual correction ratio falls below a threshold for two consecutive rounds, or when the total number of iterations reaches the upper limit, and the current annotation result is output. A new model can be used to re-label the entire dataset, but only the samples corrected in the previous round are subject to mandatory manual review; other samples are automatically approved. This process forms an automated iteration of annotation, training, and validation, which can continuously improve the model's capabilities.

[0156] This embodiment only reviews samples with high uncertainty / high contribution, reducing manpower input, ensuring labeling accuracy, dynamically triggering incremental training, and saving training resources.

[0157] The foregoing has described in detail an embodiment of a data annotation method based on a large model. Based on the data annotation method based on a large model described in the above embodiment, this invention also provides a data annotation system based on a large model corresponding to the method.

[0158] Figure 2 This is a schematic block diagram of a data annotation system based on a large model, provided as an embodiment of the present invention. In this embodiment, the data annotation system 200 based on the large model can be divided into multiple functional modules according to the functions it performs. A module, as referred to in this invention, is a series of computer program segments that can be executed by at least one processor and perform a fixed function, and is stored in memory.

[0159] The dataset acquisition module 210 is used to acquire the dataset to be labeled and to preprocess the dataset to be labeled to generate a standardized dataset.

[0160] The pre-labeling processing module 220 is used to load the fine-tuned domain model to perform batch pre-labeling on the standardized dataset and obtain the pre-labeling results of the standardized dataset.

[0161] The audit data generation module 230 is used to analyze the pre-labeled results, calculate the prediction uncertainty of each sample in the standardized dataset, select several samples based on the prediction uncertainty to form the first audit dataset, predict the contribution of each sample in the standardized dataset to the improvement of the domain model, and select several samples based on the improvement contribution to form the second audit dataset.

[0162] The manual review module 240 is used to take the union of the first dataset to be reviewed and the second dataset to be reviewed to generate the target dataset to be reviewed, and to manually review the pre-labeled results of each sample in the target dataset to be reviewed.

[0163] The annotation result generation module 250 is used to obtain the annotation results of the standardized dataset based on the results of manual review.

[0164] The large-model-based data annotation system of this embodiment is used to implement the aforementioned large-model-based data annotation method. Therefore, the specific implementation of this system can be found in the embodiment section of the large-model-based data annotation method above. Thus, the specific implementation can be referred to the description of the corresponding embodiments, and will not be elaborated here.

[0165] Furthermore, since the large-model-based data annotation system in this embodiment is used to implement the aforementioned large-model-based data annotation method, its function corresponds to that of the above method, and will not be repeated here.

[0166] Figure 3 A schematic diagram of a terminal 300 provided in an embodiment of the present invention includes: a processor 310, a memory 320, and a communication unit 330. The processor 310 is used to implement the following steps when executing the data annotation program based on a large model stored in the memory 320:

[0167] Obtain the dataset to be labeled, and preprocess the dataset to be labeled to generate a standardized dataset;

[0168] The fine-tuned domain model is loaded and pre-labeled in batches on the standardized dataset to obtain the pre-labeling results of the standardized dataset;

[0169] The pre-labeled results are analyzed, the prediction uncertainty of each sample in the standardized dataset is calculated, and a number of samples are selected based on the prediction uncertainty to form the first dataset to be reviewed.

[0170] Predict the contribution of each sample in the standardized dataset to the improvement of the domain model, and select several samples based on the improvement contribution to form a second dataset to be reviewed.

[0171] The first dataset to be reviewed and the second dataset to be reviewed are combined to generate the target dataset to be reviewed. The pre-labeled results of each sample in the target dataset to be reviewed are then manually reviewed.

[0172] The annotation results of the standardized dataset are obtained based on the results of manual review.

[0173] The terminal 300 includes a processor 310, a memory 320, and a communication unit 330. These components communicate via one or more buses. Those skilled in the art will understand that the server structure shown in the figure does not constitute a limitation of the present invention. It can be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0174] The memory 320 can be used to store the execution instructions of the processor 310. The memory 320 can be implemented by any type of volatile or non-volatile memory terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the terminal 300 is able to perform some or all of the steps in the above method embodiments.

[0175] The processor 310 serves as the control center of the storage terminal, connecting various parts of the electronic terminal via various interfaces and lines. It executes software programs and / or modules stored in the memory 320, and calls data stored in the memory to perform various functions of the electronic terminal and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 310 may consist only of a central processing unit (CPU). In this embodiment of the invention, the CPU may have a single processing core or include multiple processing cores.

[0176] The communication unit 330 is used to establish a communication channel, enabling the storage terminal to communicate with other terminals. It can receive user data sent by other terminals or send user data to other terminals.

[0177] The present invention also provides a computer storage medium, which may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0178] The present invention also provides a computer storage medium, which may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0179] The computer storage medium stores a data annotation program based on a large model. When the data annotation program based on the large model is executed by the processor, it performs the following steps:

[0180] Obtain the dataset to be labeled, and preprocess the dataset to be labeled to generate a standardized dataset;

[0181] The fine-tuned domain model is loaded and pre-labeled in batches on the standardized dataset to obtain the pre-labeling results of the standardized dataset;

[0182] The pre-labeled results are analyzed, the prediction uncertainty of each sample in the standardized dataset is calculated, and a number of samples are selected based on the prediction uncertainty to form the first dataset to be reviewed.

[0183] Predict the contribution of each sample in the standardized dataset to the improvement of the domain model, and select several samples based on the improvement contribution to form a second dataset to be reviewed.

[0184] The first dataset to be reviewed and the second dataset to be reviewed are combined to generate the target dataset to be reviewed. The pre-labeled results of each sample in the target dataset to be reviewed are then manually reviewed.

[0185] The annotation results of the standardized dataset are obtained based on the results of manual review.

[0186] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or other media capable of storing program code. It includes several instructions to cause a computer terminal (which may be a personal computer, server, or a second terminal, network terminal, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0187] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0188] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0189] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0190] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data annotation method based on a large model, characterized in that, Includes the following steps: Obtain the dataset to be labeled, and preprocess the dataset to be labeled to generate a standardized dataset; The fine-tuned domain model is loaded and pre-labeled in batches on the standardized dataset to obtain the pre-labeling results of the standardized dataset; The pre-labeled results are analyzed, the prediction uncertainty of each sample in the standardized dataset is calculated, and a number of samples are selected based on the prediction uncertainty to form the first dataset to be reviewed. The contribution of each sample in the standardized dataset to the improvement of the domain model is predicted. Based on the improvement contribution, several samples are selected to form a second dataset to be reviewed. The contribution of each sample in the standardized dataset to the improvement of the domain model includes: calculating the gradient of the domain model loss function with respect to the domain model parameters for the sample, and calculating the L2 norm of the gradient as the first initial contribution score; calculating the expected change of the domain model parameters for a single-step gradient descent for the sample, and calculating the norm of the parameter change as the second initial contribution score; calculating the density of the sample using kernel density estimation, and adjusting the first and second initial contribution scores using this density to obtain the first and second contribution scores. The first dataset to be reviewed and the second dataset to be reviewed are combined to generate the target dataset to be reviewed. The pre-labeled results of each sample in the target dataset to be reviewed are then manually reviewed. The annotation results of the standardized dataset are obtained based on the results of manual review.

2. The data annotation method based on a large model according to claim 1, characterized in that, The pre-labeled results are analyzed to calculate the prediction uncertainty for each sample in the standardized dataset, specifically including: Obtain the class probability output of the sample; The entropy value of a sample is calculated based on the category probability output using the entropy method. The confidence uncertainty score of a sample is calculated based on the category probability output using the lowest confidence method. The marginal uncertainty fraction of a sample is calculated based on the class probability output using the marginal sampling method.

3. The data annotation method based on a large model according to claim 2, characterized in that, Based on the uncertainty of prediction, a number of samples were selected to form the first dataset to be reviewed, specifically including: The sample's prediction uncertainty assessment value is obtained by weighting and summing the entropy value, confidence uncertainty score, and marginal uncertainty score. The samples in the standardized dataset are sorted in descending order of their prediction uncertainty assessment values ​​to generate the first sample sequence. Calculate the mean and standard deviation of the prediction uncertainty assessment values ​​in the first sample sequence; The first dynamic threshold is determined based on the mean and standard deviation. The first candidate set consists of samples whose prediction uncertainty assessment value is greater than or equal to the first dynamic threshold; Extract feature vectors from the samples in the first candidate set, and then use a fast clustering algorithm to cluster the samples based on the feature vectors; In each cluster, at least one sample with the highest prediction uncertainty assessment value is selected as the sample to be reviewed, and all the selected samples constitute the first dataset to be reviewed.

4. The data annotation method based on a large model according to claim 1, characterized in that, Based on the degree of improvement contribution, several samples were selected to form the second dataset to be reviewed, specifically including: The contribution evaluation value of the sample is obtained by weighted summing of the first contribution score and the second contribution score; The samples in the standardized dataset are sorted in descending order of contribution evaluation value to generate a second sample sequence; Calculate the mean and standard deviation of the contribution assessment values ​​in the second sample sequence; The second dynamic threshold is determined based on the mean and standard deviation. Samples with contribution evaluation values ​​greater than or equal to the second dynamic threshold constitute the second candidate set; Extract feature vectors from the samples in the second candidate set, and then use a fast clustering algorithm to cluster the samples based on the feature vectors; In each cluster, at least one sample with the highest contribution evaluation value is selected as the sample to be reviewed, and all the selected samples constitute the second dataset to be reviewed.

5. The data annotation method based on a large model according to claim 1, characterized in that, The annotation results of the standardized dataset are obtained based on the results of manual review, specifically including: Check whether the number of manually corrected annotations exceeds a threshold; If not, update the manually corrected results in the pre-annotation results to the manually corrected annotation results to obtain the final annotation results; If so, the corrected annotation results and corresponding samples are used to form a training set, and the domain model is retrained using this training set. The pre-labeled results of the standardized dataset are obtained again using the retrained domain model until the number of manually corrected labels is less than a threshold. Then, the manually corrected results in the current pre-labeled results are updated to the current manually corrected labels to obtain the final labeling results.

6. The data annotation method based on a large model according to claim 1, characterized in that, Preprocessing of the dataset to be labeled includes: Detect the format of the data to be labeled; If the data is in text format, then the data to be labeled will be processed by word segmentation and stop word removal. If the data is in image format, it will be normalized and resized. If the data is in audio format, noise reduction and segmentation are performed on the data to be labeled.

7. A data annotation system based on a large model, characterized in that, include: The dataset acquisition module is used to acquire the dataset to be labeled and to preprocess the dataset to be labeled to generate a standardized dataset. The pre-labeling processing module is used to load the fine-tuned domain model to perform batch pre-labeling on the standardized dataset and obtain the pre-labeling results of the standardized dataset. The audit data generation module is used to analyze the pre-labeled results, calculate the prediction uncertainty of each sample in the standardized dataset, and select several samples to form the first audit dataset based on the prediction uncertainty. It then predicts the contribution of each sample in the standardized dataset to the improvement of the domain model, and selects several samples to form the second audit dataset based on the improvement contribution. The contribution of each sample in the standardized dataset to the improvement of the domain model includes: calculating the gradient of the domain model loss function with respect to the domain model parameters for the sample, and calculating the L2 norm of this gradient as the first initial contribution score; calculating the expected change in domain model parameters for a single-step gradient descent for the sample, and calculating the norm of this parameter change as the second initial contribution score; calculating the density of the samples using kernel density estimation, and adjusting the first and second initial contribution scores using this density to obtain the first and second contribution scores. The manual review module is used to take the union of the first dataset to be reviewed and the second dataset to be reviewed to generate the target dataset to be reviewed, and to manually review the pre-labeled results of each sample in the target dataset to be reviewed. The annotation result generation module is used to obtain annotation results for standardized datasets based on the results of manual review.

8. A terminal, characterized in that, include: Memory, used to store data annotation programs based on large models; A processor, configured to implement the steps of the large-model-based data annotation method as described in any one of claims 1 to 6 when executing the large-model-based data annotation program.

9. A computer-readable storage medium, characterized in that, The readable storage medium stores a data annotation program based on a large model, which, when executed by a processor, implements the steps of the data annotation method based on a large model as described in any one of claims 1 to 6.

Images