Data annotation method and system based on large model, terminal and medium

Abstract

The invention relates to the field of data annotation, and particularly discloses a data annotation method and system based on a large model, a terminal and a medium. Loading the fine-tuned domain model to carry out batch pre-labeling on the standardized data set to obtain a pre-labeling result of the standardized data set; calculating the prediction uncertainty of each sample in the standardized data set, and selecting a plurality of samples according to the prediction uncertainty to form a first to-be-audited data set; predicting the contribution degree of each sample in the standardized data set to the improvement of the domain model, and selecting a plurality of samples according to the improvement contribution degree to form a second to-be-audited data set; taking a union set of the first to-be-audited data set and the second to-be-audited data set to generate a to-be-audited target data set, and manually auditing the pre-labeling result of each sample in the to-be-audited target data set; and obtaining a labeling result of the standardized data set according to a manual auditing result. According to the invention, the data labeling efficiency and precision are improved.

Description

Technical Field

[0001] The present invention relates to the field of data annotation, and particularly to a data annotation method, system, terminal and medium based on a large model. Background Art

[0002] Currently, artificial intelligence technology is developing rapidly. High-quality data sets are the core foundation for training and optimizing models. The accuracy and efficiency of data set annotation directly affect the performance of the model. Traditional data set annotation methods mainly rely on manual annotation. This method not only takes a lot of time and effort, has extremely high labor costs, but also has low annotation efficiency and is difficult to meet the needs of massive data annotation. With the wide application of deep learning models, model-assisted automatic annotation methods have gradually emerged. However, there are many problems in existing automatic annotation technologies, such as insufficient annotation accuracy, difficulty in adapting to data characteristics in different fields, and lack of comprehensive consideration of the reliability of annotation results and the value of model optimization. As a result, the quality of annotation results is uneven, and the role of annotated data in improving model performance cannot be fully exerted. Therefore, there is an urgent need for an efficient, accurate data annotation method that can also take into account the needs of model optimization. Summary of the Invention

[0003] To solve the above problems, the present invention provides a data annotation method, system, terminal and medium based on a large model to improve the efficiency and accuracy of data annotation.

[0004] In a first aspect, the technical solution of the present invention provides a data annotation method based on a large model, including the following steps: Obtain a data set to be annotated, and preprocess the data set to be annotated to generate a standardized data set; Load a fine-tuned domain model to perform batch pre-annotation on the standardized data set to obtain pre-annotation results of the standardized data set; Analyze the pre-annotation results, calculate the prediction uncertainty of each sample in the standardized data set, and select a number of samples according to the prediction uncertainty to form a first data set to be reviewed; Predict the contribution degree of each sample in the standardized data set to the improvement of the domain model, and select a number of samples according to the improvement contribution degree to form a second data set to be reviewed; Take the union of the first data set to be reviewed and the second data set to be reviewed to generate a target data set to be reviewed, and manually review the pre-annotation results of each sample in the target data set to be reviewed; Obtain the annotation results of the standardized data set according to the manual review results.

[0005] In an optional embodiment, analyzing the pre-annotation results and calculating the prediction uncertainty of each sample in the standardized data set specifically includes: Obtain the class probability output of the sample; Calculate the entropy value of the sample according to the category probability output by the entropy method; Calculate the confidence uncertainty score of the sample according to the category probability output by the lowest confidence method; Calculate the marginal uncertainty score of the sample according to the category probability output by the marginal sampling method.

[0006] In an optional implementation manner, several samples are selected to form a first dataset to be reviewed according to the prediction uncertainty, specifically including: Perform weighted summation on the entropy value, confidence uncertainty score, and marginal uncertainty score of the sample to obtain the prediction uncertainty evaluation value of the sample; Sort each sample in the standardized dataset in descending order according to the prediction uncertainty evaluation value to generate a first sample sequence; Calculate the mean and standard deviation of the prediction uncertainty evaluation values in the first sample sequence; Determine a first dynamic threshold according to the mean and standard deviation; Form a first candidate set with samples whose prediction uncertainty evaluation value is greater than or equal to the first dynamic threshold; Extract the feature vectors of the samples in the first candidate set, and perform sample clustering on the samples using the fast clustering algorithm; Select at least one sample with the highest prediction uncertainty evaluation value in each cluster as the sample to be reviewed, and all the selected samples form the first dataset to be reviewed.

[0007] In an optional implementation manner, predict the contribution degree of each sample in the standardized dataset to the improvement of the domain model, specifically including: Calculate the gradient of the domain model loss function with respect to the domain model parameters for the sample, and calculate the L2 norm of the gradient as the first initial contribution score; Calculate the expected change in the domain model parameters of the single-step gradient descent for the sample, and calculate the norm of the parameter change as the second initial contribution score; Use kernel density estimation to calculate the density of the sample, and use the density to adjust the first initial contribution score and the second initial contribution score to obtain the first contribution score and the second contribution score.

[0008] In an optional implementation manner, several samples are selected to form a second dataset to be reviewed according to the improvement contribution degree, specifically including: Perform weighted summation on the first contribution score and the second contribution score of the sample to obtain the contribution degree evaluation value of the sample; Sort each sample in the standardized dataset in descending order according to the contribution degree evaluation value to generate a second sample sequence; Calculate the mean and standard deviation of the contribution degree evaluation values in the second sample sequence; Determine the second dynamic threshold according to the mean value and the standard deviation; Construct a second candidate set with samples whose contribution degree evaluation value is greater than or equal to the second dynamic threshold; Extract feature vectors from the samples in the second candidate set, and perform sample clustering on the samples according to the feature vectors using a fast clustering algorithm; Select at least one sample with the highest contribution degree evaluation value in each cluster as the sample to be reviewed, and all the selected samples form a second dataset to be reviewed.

[0009] In an alternative embodiment, obtain the annotation result of the standardized dataset according to the manual review result, specifically including: Detect whether the number of manually corrected annotation results exceeds the threshold; If not, update the manually corrected results in the pre-annotation results to the manually corrected annotation results to obtain the final annotation result; If so, form a training set with the corrected annotation results and the corresponding samples, and retrain the domain model through this training set; Use the retrained domain model to obtain the pre-annotation results of the standardized dataset again until the number of manually corrected annotation results is less than the threshold, and update the manually corrected results in the current pre-annotation results to the current manually corrected annotation results to obtain the final annotation result.

[0010] In an alternative embodiment, the preprocessing of the dataset to be annotated includes: Detect the format of the data to be annotated; If it is in text format, perform word segmentation and stop word removal on the data to be annotated; If it is in picture format, perform normalization processing and size adjustment processing on the data to be annotated; If it is in audio format, perform noise reduction processing and segmentation processing on the data to be annotated.

[0011] In a second aspect, the technical solution of the present invention provides a data annotation system based on a large model, including: A dataset acquisition module, configured to acquire a dataset to be annotated and perform preprocessing on the dataset to be annotated to generate a standardized dataset; A pre-annotation processing module, configured to load the fine-tuned domain model to perform batch pre-annotation on the standardized dataset to obtain the pre-annotation results of the standardized dataset; An audit data generation module, configured to analyze the pre-annotation results, calculate the prediction uncertainty of each sample in the standardized dataset, and select several samples to form a first dataset to be audited according to the prediction uncertainty; predict the contribution degree of each sample in the standardized dataset to the improvement of the domain model, and select several samples to form a second dataset to be audited according to the improvement contribution degree; A manual review module is used to combine the first dataset to be reviewed and the second dataset to be reviewed to generate a target dataset to be reviewed, and to manually review the pre-labeled results of each sample in the target dataset to be reviewed; The annotation result generation module is used to obtain the annotation results of the standardized data set based on the manual review results.

[0012] In a third aspect, the technical solution of the present invention provides a terminal, including: Memory, used to store data annotation programs based on large models; A processor is used to implement the steps of the data labeling method based on a large model as described in any of the above items when executing the data labeling program based on the large model.

[0013] In a fourth aspect, the technical solution of the present invention provides a computer-readable storage medium, on which a data labeling program based on a large model is stored. When the data labeling program based on a large model is executed by a processor, the steps of the data labeling method based on a large model as described in any of the above items are implemented.

[0014] It can be seen from the above technical solutions that the present application has the following advantages: first, the data is pre-labeled using the fine-tuned domain model, and then the pre-labeled results of several samples are selected as the results to be reviewed based on the prediction uncertainty of each sample and the contribution to the improvement of the domain model, and manually reviewed, and finally the labeling results of the standardized data set are obtained based on the manual review results. The present invention improves the labeling efficiency, reduces the workload of manual labeling, and reduces the labeling cost by fine-tuning the domain model and using it for batch pre-labeling. Further, in the pre-labeling result processing link, samples are selected by calculating the prediction uncertainty to form the first data set to be reviewed, and samples with low reliability of the labeling results are accurately screened out, ensuring that the manual review focuses on key data and improving the labeling accuracy; and based on the contribution of the samples to the improvement of the domain model, the second data set to be reviewed is selected, which can mine samples of great value for model optimization, achieve the fit between the labeled data and the model training, and effectively improve the model performance. By taking the union of the two data sets to be reviewed to generate the target data set to be reviewed, a targeted manual review is performed, which not only ensures the labeling quality, but also improves the review efficiency, and finally obtains high-quality labeling results, further improves the labeling accuracy, and provides high-quality data support for subsequent model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for the description. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0016] Figure 1 A schematic flowchart of a data annotation method based on a large model provided by an embodiment of the present invention.

[0017] Figure 2 A schematic block diagram of the structure of a data annotation system based on a large model provided by an embodiment of the present invention.

[0018] Figure 3 A schematic diagram of the structure of a terminal provided by an embodiment of the present invention. Detailed implementation manners

[0019] To make the application purpose, features, and advantages of the present application more obvious and understandable, the technical solutions protected by the present application will be clearly and completely described below by using specific embodiments and the accompanying drawings. Obviously, the embodiments described below are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0020] Unless otherwise defined, all technical and scientific terms used in the present application have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs. The terms used in the description of the present invention in this application are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0021] Figure 1 A schematic flowchart of a data annotation method based on a large model provided by an embodiment of the present invention. Among them, Figure 1 The execution subject can be a data annotation system based on a large model. The data annotation method based on a large model provided by an embodiment of the present invention is executed by a computer device. Correspondingly, the data annotation system based on a large model runs in the computer device. According to different requirements, the order of the steps in this flowchart can be changed, and some can be omitted.

[0022] Such as Figure 1 shown, the method includes the following steps.

[0023] S1, obtain the dataset to be annotated, and preprocess the dataset to be annotated to generate a standardized dataset.

[0024] Obtain the dataset to be annotated, and perform preprocessing operations such as word segmentation and stop word removal, normalized size adjustment, noise reduction and segmentation on data in different formats such as text, pictures, and audio to generate a standardized dataset.

[0025] Specifically, the format of the data to be annotated is detected; if it is in text format, the annotated data is segmented and stop words are removed; if it is in image format, the annotated data is normalized and resized; if it is in audio format, the annotated data is denoised and segmented.

[0026] In some optional implementations, multimodal data to be annotated, such as text, images, and audio, are obtained through interface calls, file imports, etc., supporting the input of structured (such as tables) and unstructured data (such as documents and video frames).

[0027] For text data, we use word segmentation tools to split sentences into word vectors, and use stop word lists to filter out meaningless words such as "的" and "了". At the same time, we perform normalization processing such as unifying uppercase and lowercase letters and cleaning special symbols.

[0028] For image data, use the OpenCV library to normalize pixel values (scale to the [0,1] interval), adjust the size according to the model input requirements, and perform grayscale conversion or channel standardization.

[0029] For audio data, Fourier transform is used for noise reduction, the audio clips are segmented through the VAD (Voice Activity Detection) algorithm, and feature vectors such as Mel-Frequency Cepstral Coefficients (MFCC) are extracted.

[0030] Unify data in different formats into an input format recognizable by the model, such as converting text into ID sequences, images into tensors, and audio into feature matrices, and store them in standard data formats such as JSON or TFRecord.

[0031] S2, load the fine-tuned domain model to perform batch pre-labeling on the standardized dataset to obtain the pre-labeling results of the standardized dataset.

[0032] Load the fine-tuned domain model and perform batch pre-annotation on the standardized dataset to obtain the pre-annotated results. Specifically, based on the general large model, supervised fine-tuning is first performed using a small amount of annotated data within the domain, and the optimization objective function is cross-entropy loss or mean squared error. Then, for professional fields such as medicine and law, domain dictionaries and knowledge graphs are introduced to adjust parameter weights to improve the recognition of professional terminology.

[0033] A batch processing mechanism is adopted, GPU parallel computing is used to accelerate inference, and the category probability distribution or regression value of each sample is generated through forward propagation of the model. When outputting the pre-labeled results, the confidence score (such as softmax probability) of each predicted category is recorded to provide a data basis for subsequent uncertainty analysis.

[0034] S3. Analyze the pre-annotation results, calculate the prediction uncertainty of each sample in the standardized dataset, and select a number of samples based on the prediction uncertainty to form the first dataset to be reviewed.

[0035] Analyze the pre-annotation results, calculate the relevant indicators of the prediction uncertainty of the samples respectively by the entropy method, the lowest confidence method, and the marginal sampling method, sum up these indicators after weighting to obtain the prediction uncertainty evaluation value, determine the dynamic threshold by combining the mean and standard deviation after sorting according to this, and screen out the samples with high uncertainty to form a candidate set. Cluster the samples in the candidate set, and select the sample with the highest prediction uncertainty in each category to form the first dataset to be reviewed.

[0036] S4. Predict the contribution degree of each sample in the standardized dataset to the improvement of the domain model, and select a number of samples based on the improvement contribution degree to form the second dataset to be reviewed.

[0037] Calculate the contribution degree of the sample to the improvement of the domain model, including the first initial contribution degree score based on the L2 norm of the loss function gradient and the second initial contribution degree score of the expected parameter change norm of the single-step gradient descent, and adjust these two scores using kernel density. Sum up the adjusted contribution degree scores after weighting to obtain the contribution degree evaluation value, determine the dynamic threshold by combining the mean and standard deviation after sorting according to this, and screen out the samples with high contribution degree to form a candidate set. Cluster the samples in the candidate set, and select the sample with the highest contribution degree in each category to form the second dataset to be reviewed.

[0038] S5. Take the union of the first dataset to be reviewed and the second dataset to be reviewed to generate the target dataset to be reviewed, and conduct manual review on the pre-annotation results of each sample in the target dataset to be reviewed.

[0039] Take the union of the first and second datasets to be reviewed, remove duplicates by hash value to avoid repeated review of the same data. It can be grouped by sample type (text / image / audio) and domain label (such as lungs / heart in medical imaging) to facilitate the division of labor for reviewers.

[0040] Provide a visual review interface to display the pre-annotation results, model confidence, uncertainty indicators, and contribution degree scores to assist reviewers in making judgments. Support functions such as batch annotation, annotation history backtracking, and marking of controversial samples. The review results are stored in JSON format in real time, including the original data, pre-annotation results, manual correction values, and review timestamps.

[0041] S6. Obtain the annotation results of the standardized dataset according to the manual review results.

[0042] Count the number of manually corrected annotations. If the correction rate (number of corrections / total number of reviews) is lower than the preset threshold (e.g., 10%), directly replace the error items in the pre-annotated results with the manually corrected values to generate the final annotated dataset. If the correction rate exceeds the threshold, form a new training set with the corrected samples and their labels, and use transfer learning to retrain the domain model (freeze the base layer and fine-tune the task layer), repeating the S2 - S6 process until the correction rate meets the standard.

[0043] In some alternative embodiments, in step S3, the pre-annotated results are analyzed to calculate the prediction uncertainty of each sample in the standardized dataset, which specifically includes the following steps.

[0044] S31.1, Obtain the class probability output of the sample.

[0045] Obtain the class probability output of the sample from the domain model , where represents the th class, .

[0046] S31.2, Calculate the entropy value of the sample according to the class probability output by the entropy method.

[0047] Calculate the entropy value of the probability distribution output by the model. The higher the entropy value, the greater the uncertainty.

[0048] The entropy value calculation formula is,

[0049] In the formula, is the uncertainty of the sample , that is, the entropy value (in bits), is the probability that the domain model predicts the sample belongs to the class , is the total number of classes.

[0050] S31.3, Calculate the confidence uncertainty score of the sample according to the class probability output by the lowest confidence method.

[0051] Select the sample with the lowest predicted probability of the model, such as the sample with the highest probability class confidence close to 0.5 in the classification task. Directly take the complement of the maximum probability predicted by the model. The larger the value, the more uncertain.

[0052] The confidence uncertainty score is calculated by the formula,

[0053] S31.4, Calculate the marginal uncertainty score of the sample according to the class probability output by the marginal sampling method.

[0054] Calculate the probability difference between the first two most likely classes predicted by the computational model. The smaller the difference, the higher the uncertainty.

[0055] Specifically, for the probability distribution sort in descending order by value to obtain the first two highest probabilities and .

[0056] Boundary uncertainty score is calculated as

[0057] Furthermore, in step S3, a number of samples are selected according to the prediction uncertainty to form the first dataset to be reviewed, which specifically includes the following steps.

[0058] S32.1, perform a weighted sum of the entropy value, confidence uncertainty score, and marginal uncertainty score of the sample to obtain the prediction uncertainty evaluation value of the sample.

[0059] S32.2, sort each sample in the standardized dataset in descending order according to the prediction uncertainty evaluation value to generate the first sample sequence.

[0060] S32.3, calculate the mean and standard deviation of the prediction uncertainty evaluation values in the first sample sequence.

[0061] S32.4, determine the first dynamic threshold according to the mean and standard deviation.

[0062] Specifically, the calculation formula for the first dynamic threshold is

[0063] where is the first dynamic threshold, is the mean calculated in step S32.3, is the standard deviation calculated in step S32.3, is the adjustment coefficient.

[0064] The first dynamic threshold is automatically adjusted according to the data distribution, avoiding inefficiency caused by fixed ratio selection.

[0065] S32.5, form the first candidate set with samples whose prediction uncertainty evaluation values are greater than or equal to the first dynamic threshold.

[0066] S32.6, extract the feature vectors of the samples in the first candidate set, and perform sample clustering on the feature vectors using the fast clustering algorithm.

[0067] For example, extract BERT text embeddings and CNN image features from the first candidate set. Clustering ensures coverage of different data patterns and avoids duplicate annotation of similar samples.

[0068] S32.7, select at least one sample with the highest prediction uncertainty evaluation value in each cluster as the sample to be reviewed, and all the selected samples form the first dataset to be reviewed.

[0069] Select the sample with the highest ranking value from each cluster to ensure coverage of different data distribution patterns. If the proportion of samples in a certain cluster exceeds 30%, the sample with the second highest ranking value in that cluster can be additionally selected to prevent large clusters from dominating.

[0070] In some alternative embodiments, the contribution degree of each sample in the predicted standardized dataset to the improvement of the domain model in step S4 specifically includes the following steps.

[0071] S41.1, calculate the gradient of the domain model loss function with respect to the domain model parameters for the sample, and calculate the L2 norm of this gradient as the first initial contribution score.

[0072] The greater the gradient of the sample with respect to the model loss function, the more significant its impact on parameter update and the higher the contribution degree.

[0073] Let the domain model parameters be , and the loss function be , where is the sample, and

[0074] For the sample calculate the gradient of the loss function with respect to the parameters:

[0075] Calculate the gradient norm, and the L2 norm of the gradient is used as the first initial contribution score :

[0076] In the formula, is the dimension of the domain model parameters, is the gradient component of the

[0077] S41.2, calculate the expected change in the domain model parameters for a single-step gradient descent for the sample, and calculate the norm of this parameter change as the second initial contribution score.

[0078] After predicting the labeled sample, the amplitude of the model parameter update, and the greater the amplitude, the higher the contribution degree.

[0079] Assume that the true label of the labeled sample is , calculate the expected parameter change of single-step gradient descent:

[0080] wherein, is the learning rate.

[0081] Calculate the change norm as the second initial contribution score :

[0082] S41.3, use kernel density estimation to calculate the density of the samples, and use this density to adjust the first initial contribution score and the second initial contribution score to obtain the first contribution score and the second contribution score.

[0083] Use kernel density estimation (KDE) to calculate the density of the samples :

[0084] wherein, is the kernel function, is the bandwidth parameter, is the number of samples.

[0085] The first contribution score ; The second contribution score ; wherein is the adjustment parameter.

[0086] Furthermore, in step S4, several samples are selected from the standardized dataset according to the improved contribution degree to form the second dataset to be reviewed, which specifically includes the following steps.

[0087] S42.1, perform weighted summation on the first contribution score and the second contribution score of the samples to obtain the contribution degree evaluation value of the samples.

[0088] S42.2, sort each sample in the standardized dataset in descending order according to the contribution degree evaluation value to generate the second sample sequence.

[0089] S42.3, calculate the mean and standard deviation of the contribution degree evaluation values in the second sample sequence.

[0090] S42.4, determine the second dynamic threshold according to the mean and the standard deviation.

[0091] S42.5, form the second candidate set with the samples whose contribution degree evaluation values are greater than or equal to the second dynamic threshold.

[0092] S42.6, extract the feature vectors of the samples in the second candidate set, and perform sample clustering on the samples according to the feature vectors using the fast clustering algorithm.

[0093] S42.7, select at least one sample with the highest contribution evaluation value in each cluster as the sample to be reviewed, and all the selected samples constitute the second dataset to be reviewed.

[0094] This process is similar to steps S32.1 to S32.7 and will not be elaborated here.

[0095] In some alternative embodiments, step S6 obtains the annotation result of the standardized dataset according to the manual review result, which specifically includes the following steps.

[0096] S61, detect whether the number of manually corrected annotation results exceeds the threshold.

[0097] Specifically, a proportional threshold or an absolute threshold can be set according to the size of the dataset.

[0098] S62, if not, update the manually corrected results in the pre-annotation results to the manually corrected annotation results to obtain the final annotation result.

[0099] For the uncorrected samples, retain the pre-annotation results. For the corrected samples, update them to the manual labels, so as to directly output the results during small-scale correction and shorten the annotation cycle. For large-scale correction, it is necessary to retrain the domain model, such as step S63.

[0100] S63, if so, form a training set with the corrected annotation results and the corresponding samples, and retrain the domain model through this training set.

[0101] The corrected samples can be directly used for retraining, or the corrected samples can be added to the current training samples to form a total sample for retraining, so as to achieve enhanced training for the areas where the model makes mistakes and improve the efficiency.

[0102] S64, use the retrained domain model to obtain the pre-annotation results of the standardized dataset again until the number of manually corrected annotation results is less than the threshold, and update the manually corrected results in the current pre-annotation results to the current manually corrected annotation results to obtain the final annotation result.

[0103] Specifically, when the manual correction ratio is lower than the threshold for two consecutive rounds, or the total number of iterations reaches the upper limit, the loop is terminated and the current annotation result is output. The new model can be used to re-pre-annotate all the data, but only the corrected samples in the previous round are forced to be manually reviewed, and the other samples pass automatically. This process forms an automated iteration of annotation, training, and verification, which can continuously improve the model's ability.

[0104] This embodiment only reviews samples with high uncertainty / high contribution, reduces the human input, ensures the annotation accuracy, dynamically triggers incremental training, and saves training resources.

[0105] In the foregoing, embodiments of a large model-based data annotation method have been described in detail. Based on the large model-based data annotation method described in the foregoing embodiments, embodiments of the present invention also provide a large model-based data annotation system corresponding to this method.

[0106] Figure 2 FIG. is a schematic block diagram of the structure of a large model-based data annotation system provided by an embodiment of the present invention. In this embodiment, the large model-based data annotation system 200 can be divided into multiple functional modules according to the functions it performs. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, and are stored in the memory.

[0107] The dataset acquisition module 210 is used to acquire the dataset to be annotated and preprocess the dataset to be annotated to generate a standardized dataset.

[0108] The pre-annotation processing module 220 is used to load the fine-tuned domain model to batch pre-annotate the standardized dataset and obtain the pre-annotation results of the standardized dataset.

[0109] The review data generation module 230 is used to analyze the pre-annotation results, calculate the prediction uncertainty of each sample in the standardized dataset, and select a number of samples to form the first dataset to be reviewed according to the prediction uncertainty; predict the contribution degree of each sample in the standardized dataset to the improvement of the domain model, and select a number of samples to form the second dataset to be reviewed according to the improvement contribution degree.

[0110] The manual review module 240 is used to take the union of the first dataset to be reviewed and the second dataset to be reviewed to generate the target dataset to be reviewed, and manually review the pre-annotation results of each sample in the target dataset to be reviewed.

[0111] The annotation result generation module 250 is used to obtain the annotation results of the standardized dataset according to the manual review results.

[0112] The large model-based data annotation system of this embodiment is used to implement the foregoing large model-based data annotation method. Therefore, the specific implementation manners in this system can be seen in the embodiment part of the large model-based data annotation method in the foregoing text. Therefore, its specific implementation manners can be referred to the descriptions of the corresponding parts of the embodiments, and will not be elaborated here.

[0113] In addition, since the large model-based data annotation system of this embodiment is used to implement the foregoing large model-based data annotation method, its functions correspond to those of the above method, and will not be elaborated here.

[0114] Figure 3Schematic diagram of a structure of a terminal 300 provided by an embodiment of the present invention, including: a processor 310, a memory 320, and a communication unit 330. When the processor 310 is used to implement the data annotation program based on a large model stored in the memory 320, the following steps are implemented: Obtain a dataset to be annotated, and preprocess the dataset to be annotated to generate a standardized dataset; Load the fine-tuned domain model to batch pre-annotate the standardized dataset, and obtain the pre-annotation result of the standardized dataset; Analyze the pre-annotation result, calculate the prediction uncertainty of each sample in the standardized dataset, and select several samples according to the prediction uncertainty to form a first dataset to be reviewed; Predict the contribution degree of each sample in the standardized dataset to the improvement of the domain model, and select several samples according to the improvement contribution degree to form a second dataset to be reviewed; Take the union of the first dataset to be reviewed and the second dataset to be reviewed to generate a target dataset to be reviewed, and perform manual review on the pre-annotation results of each sample in the target dataset to be reviewed; Obtain the annotation result of the standardized dataset according to the manual review result.

[0115] The terminal 300 includes a processor 310, a memory 320, and a communication unit 330. These components communicate through one or more buses. Those skilled in the art can understand that the structure of the server shown in the figure does not constitute a limitation to the present invention. It can be a bus structure, a star structure, and can also include more or fewer components than shown in the figure, or combine some components, or arrange different components.

[0116] Among them, the memory 320 can be used to store the execution instructions of the processor 310. The memory 320 can be implemented by any type of volatile or non-volatile storage terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the terminal 300 can execute some or all of the steps in the above method embodiments.

[0117] The processor 310 is the control center of the storage terminal, connecting various parts of the entire electronic terminal through various interfaces and circuits. By running or executing software programs and / or modules stored in the memory 320, and by invoking data stored in the memory, it executes various functions of the electronic terminal and / or processes data. The processor may be composed of an integrated circuit (IC), for example, it may be composed of a single packaged IC, or it may be composed of multiple packaged ICs with the same or different functions connected together. For example, the processor 310 may include only a central processing unit (CPU). In the embodiments of the present invention, the CPU may be a single arithmetic core or may include multiple arithmetic cores.

[0118] The communication unit 330 is used to establish a communication channel so that the storage terminal can communicate with other terminals. It receives user data sent by other terminals or sends user data to other terminals.

[0119] The present invention also provides a computer storage medium, and the storage medium here may be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), etc.

[0120] The present invention also provides a computer storage medium, and the storage medium here may be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), etc.

[0121] The computer storage medium stores a data annotation program based on a large model. When the data annotation program based on the large model is executed by the processor, the following steps are implemented: Obtain the dataset to be annotated, and perform preprocessing on the dataset to be annotated to generate a standardized dataset; Load the fine-tuned domain model to perform batch pre-annotation on the standardized dataset to obtain the pre-annotation results of the standardized dataset; Analyze the pre-annotation results, calculate the prediction uncertainty of each sample in the standardized dataset, and select several samples according to the prediction uncertainty to form the first dataset to be reviewed; Predict the contribution degree of each sample in the standardized dataset to the improvement of the domain model, and select several samples according to the improvement contribution degree to form the second dataset to be reviewed; Generate a target dataset to be reviewed by taking the union of the first dataset to be reviewed and the second dataset to be reviewed, and manually review the pre-annotation results of each sample in the target dataset to be reviewed; Obtain the annotation results of the standardized dataset according to the manual review results.

[0122] Those skilled in the art can clearly understand that the technologies in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solutions in the embodiments of the present invention, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disc, etc., which can store program codes, including several instructions to enable a computer terminal (which can be a personal computer, a server, or a second terminal, a network terminal, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0123] In several embodiments provided by the present invention, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0124] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0125] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit.

[0126] The foregoing description of the disclosed embodiments enables those skilled in the art to practice or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for data annotation based on large models, characterized in that, It includes the following steps: Obtain the dataset to be labeled, and preprocess the dataset to be labeled to generate a standardized dataset; Load the fine-tuned domain model to perform batch pre-labeling on the standardized dataset, and obtain the pre-labeling results of the standardized dataset; Analyze the pre-labeling results, calculate the prediction uncertainty of each sample in the standardized dataset, and select a number of samples according to the prediction uncertainty to form the first dataset to be reviewed; Predict the contribution degree of each sample in the standardized dataset to the improvement of the domain model, and select a number of samples according to the improvement contribution degree to form the second dataset to be reviewed; Take the union of the first dataset to be reviewed and the second dataset to be reviewed to generate the target dataset to be reviewed, and manually review the pre-labeling results of each sample in the target dataset to be reviewed; Obtain the labeling results of the standardized dataset according to the manual review results.

2. The data annotation method based on a large model according to claim 1, wherein Analyze the pre-labeling results, calculate the prediction uncertainty of each sample in the standardized dataset, specifically including: Obtain the class probability output of the sample; Calculate the entropy value of the sample according to the class probability output by the entropy method; Calculate the confidence uncertainty score of the sample according to the class probability output by the lowest confidence method; Calculate the marginal uncertainty score of the sample according to the class probability output by the marginal sampling method.

3. The data annotation method based on a large model according to claim 2, wherein Select a number of samples according to the prediction uncertainty to form the first dataset to be reviewed, specifically including: Perform weighted summation on the entropy value, confidence uncertainty score, and marginal uncertainty score of the sample to obtain the prediction uncertainty evaluation value of the sample; Sort each sample in the standardized dataset in descending order according to the prediction uncertainty evaluation value to generate the first sample sequence; Calculate the mean and standard deviation of the prediction uncertainty evaluation values in the first sample sequence; Determine the first dynamic threshold according to the mean and standard deviation; Form the first candidate set with samples whose prediction uncertainty evaluation value is greater than or equal to the first dynamic threshold; Extract the feature vectors of the samples in the first candidate set, and perform sample clustering on the samples according to the feature vectors using the fast clustering algorithm; Select at least one sample with the highest prediction uncertainty evaluation value in each cluster as the sample to be reviewed, and all the selected samples form the first dataset to be reviewed.

4. The data annotation method based on a large model according to claim 1, wherein Predict the contribution degree of each sample in the standardized dataset to the improvement of the domain model, specifically including: Calculate the gradient of the domain model loss function with respect to the domain model parameters for the sample, and calculate the L2 norm of the gradient as the first initial contribution score; Calculate the expected change in the domain model parameters of the single-step gradient descent for the sample, and calculate the norm of the parameter change as the second initial contribution score; Use kernel density estimation to calculate the density of the sample, and use the density to adjust the first initial contribution score and the second initial contribution score to obtain the first contribution score and the second contribution score.

5. The data annotation method based on a large model according to claim 4, wherein Select a number of samples according to the improvement contribution degree to form the second dataset to be reviewed, specifically including: Perform weighted summation on the first contribution score and the second contribution score of the sample to obtain the contribution degree evaluation value of the sample; Sort each sample in the standardized dataset in descending order according to the contribution degree evaluation value to generate the second sample sequence; Calculate the mean and standard deviation of the contribution degree evaluation values in the second sample sequence; Determine the second dynamic threshold based on the mean value and the standard deviation; Form a second candidate set with the samples whose contribution degree evaluation values are greater than or equal to the second dynamic threshold; Extract feature vectors from the samples in the second candidate set, and perform sample clustering on the samples using a fast clustering algorithm according to the feature vectors; Select at least one sample with the highest contribution degree evaluation value in each cluster as the sample to be audited, and all the selected samples form a second dataset to be audited.

6. The data annotation method based on a large model according to claim 1, wherein Obtain the annotation results of the standardized dataset according to the manual audit results, specifically including: Detect whether the number of manually corrected annotation results exceeds the threshold; If not, update the manually corrected results in the pre-annotation results to the manually corrected annotation results to obtain the final annotation results; If so, form a training set with the corrected annotation results and the corresponding samples, and retrain the domain model through this training set; Use the retrained domain model to obtain the pre-annotation results of the standardized dataset again until the number of manually corrected annotation results is less than the threshold, and update the manually corrected results in the current pre-annotation results to the current manually corrected annotation results to obtain the final annotation results.

7. The data annotation method based on a large model according to claim 1, wherein The preprocessing of the dataset to be annotated includes: Detect the format of the data to be annotated; If it is in text format, perform word segmentation and stop word removal on the data to be annotated; If it is in image format, perform normalization processing and size adjustment processing on the data to be annotated; If it is in audio format, perform noise reduction processing and segmentation processing on the data to be annotated.

8. A data annotation system based on a large model, characterized in that, Include: A dataset acquisition module, configured to acquire a dataset to be annotated, and perform preprocessing on the dataset to be annotated to generate a standardized dataset; A pre-annotation processing module, configured to load the fine-tuned domain model to perform batch pre-annotation on the standardized dataset to obtain the pre-annotation results of the standardized dataset; An audit data generation module, configured to analyze the pre-annotation results, calculate the prediction uncertainty of each sample in the standardized dataset, and select several samples to form a first dataset to be audited according to the prediction uncertainty; predict the contribution degree of each sample in the standardized dataset to the improvement of the domain model, and select several samples to form a second dataset to be audited according to the improvement contribution degree; A manual audit module, configured to take the union of the first dataset to be audited and the second dataset to be audited to generate a target dataset to be audited, and perform manual audit on the pre-annotation results of each sample in the target dataset to be audited; An annotation result generation module, configured to obtain the annotation results of the standardized dataset according to the manual audit results.

9. A terminal, characterized in that, Include: A memory, configured to store a data annotation program based on a large model; A processor, configured to implement the steps of the data annotation method based on a large model according to any one of claims 1 to 7 when executing the data annotation program based on a large model.

10. A computer-readable storage medium, characterized in that, The data annotation program based on a large model is stored on the readable storage medium, and when the data annotation program based on a large model is executed by the processor, the steps of the data annotation method based on a large model according to any one of claims 1 to 7 are implemented.

Images