Annotated data quality assessment method based on active learning

Abstract

The present invention provides a method for evaluating the quality of labeled data based on active learning, and relates to the technical field of data collection. The method comprises: obtaining historical labeled data and historical usage scores, screening the historical labeled data according to the historical usage scores, obtaining high-quality labeled data and constructing a labeling quality evaluation model, analyzing the type of labeled data that annotators are good at, assigning unlabeled task data to the annotators, obtaining a real-time labeled data set, inputting the real-time labeled data set into the labeling quality evaluation model, obtaining a first quality score, interactively verifying the real-time labeled data set, obtaining a second quality score, obtaining a deviation value by comparison, and performing a preliminary adjustment on the labeling quality evaluation model when the deviation value is less than a preset threshold value, and performing a secondary adjustment on the labeling quality evaluation model after receiving the usage score of the real-time labeled data set. The present invention can improve the quality evaluation method.

Description

Technical Field

[0001] The present invention relates to the technical field of data collection, and in particular to a method for evaluating the quality of labeled data based on active learning. Background Art

[0002] In the prior art, the quality assessment method of labeled data is widely used in the fields of machine learning and data mining. It mainly obtains the historical labeled data of the annotator and uses the scores to train and optimize the evaluation model so as to assign more reasonable tasks to the annotator. However, this method often relies on a single quality assessment standard, that is, the score. Since the feedback may take a long time, such as a month, this leads to poor self-improvement ability of the model, slow optimization speed, and inability to quickly respond to changes in the labeling quality.

[0003] In practical applications, if the annotation style or ability of the annotator changes, a single usage score cannot reflect his latest performance in a timely manner, which may lead to a lag in the evaluation of the newly annotated data, thus affecting the quality assessment of the entire dataset. This limitation not only affects the model's accurate assessment of the annotation quality, but also reduces the efficiency of task allocation to a certain extent, thus affecting the progress and effect of the entire project. Summary of the invention

[0004] The purpose of the present invention is to provide a method for evaluating the quality of labeled data based on active learning, aiming to solve the problems mentioned in the background technology.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0006] A method for evaluating the quality of labeled data based on active learning, the method comprising:

[0007] Step 11, obtaining the annotator's historical annotation data and historical usage scores, and screening the historical annotation data according to the historical usage scores to obtain high-quality annotation data and its high-quality usage scores;

[0008] Step 12: construct an annotation quality assessment model based on the high-quality annotation data and the high-quality usage score;

[0009] Step 13, analyzing the type of annotation data that the annotator is good at based on the high-quality usage score, and obtaining the data that the annotator is good at;

[0010] Step 14, obtaining unlabeled task data, and assigning the task data to annotators based on the data that the annotators are good at, and the annotators annotate it to obtain a real-time labeled data set;

[0011] Step 15, inputting the real-time annotation data set into the annotation quality assessment model to obtain a first quality score;

[0012] Step 16, interactively verifying the annotated data in the real-time annotated data set, calculating the consistency of the annotated data, and obtaining a second quality score;

[0013] Step 17, comparing the first quality score with the second quality score to obtain a deviation value;

[0014] Step 18: when the deviation value is less than a preset threshold, the annotation quality assessment model is preliminarily adjusted according to the real-time annotation data set;

[0015] Step 19, after receiving the usage score of the real-time annotation data set, the annotation quality assessment model is adjusted again by using the usage score.

[0016] Furthermore, based on high-quality annotation data and high-quality usage scores, an annotation quality assessment model is constructed, including:

[0017] Preprocess the high-quality labeled data to obtain standardized data, and divide the standardized data into training data set and test data set through stratified sampling;

[0018] Based on the long short-term memory network model, the first annotation quality assessment model is constructed;

[0019] Inputting the training data set into the first annotation quality assessment model to obtain a training quality score;

[0020] The difference between the training quality score and the high-quality usage score is calculated through the loss function to obtain the training difference;

[0021] According to the training difference, adjusting the parameters of the first annotation quality assessment model;

[0022] When the training difference is lower than the preset threshold and the rate of change of its downward trend meets the preset rate of change, the training is stopped to obtain a preliminary labeling quality assessment model;

[0023] Input the test data set into the preliminary annotation quality assessment model to obtain the test quality score;

[0024] The difference between the test quality score and the high-quality usage score is calculated through the loss function to obtain the test difference;

[0025] When the test difference is within the preset threshold, the annotation quality assessment model is obtained. When the test difference is not within the preset threshold, retraining is performed and the direction of the adjustment parameters is modified.

[0026] Furthermore, based on the long short-term memory network model, a first annotation quality assessment model is constructed, including:

[0027] Configure the input layer to receive the input labeled data and obtain the input data;

[0028] Configure the feature extraction layer to extract feature vectors of input data and high-quality labeled data to obtain a vector set;

[0029] Configure a quality assessment layer to process the vector set and generate a quality score for the input data;

[0030] Configure the output layer to output the quality score of the input data;

[0031] A first annotation quality assessment model is constructed according to the input layer, the feature extraction layer, the quality assessment layer and the output layer.

[0032] Furthermore, the vector set is processed to generate a quality score of the input data, including:

[0033] Perform linear transformation on the vector set and calculate the intermediate vector;

[0034] The intermediate vector is activated by a nonlinear activation function to obtain an activation vector;

[0035] Mapping activation vectors to usage scores of good labeled data produces quality scores for the input data.

[0036] Furthermore, the loss function is calculated as:

[0037] ,

[0038] in, is the loss function, The training data set The training quality score of the labeled data, The training data set The usage score of the high-quality labeled data corresponding to the labeled data, is the total number of labeled data in the training dataset, is the regularization coefficient, is the L2 regularization term of the first labeled data quality assessment model parameters, The training data set The variance of the quality score of the labeled data, is the coefficient.

[0039] Further, according to the training difference, the parameters of the first annotation quality assessment model are adjusted, including:

[0040] According to the size and direction of the training difference, the gradient of the loss function relative to the first labeled data quality assessment model parameters is obtained;

[0041] The gradients of all parameters in the first annotation quality assessment model are obtained by the chain rule to generate gradient change information;

[0042] According to the gradient change information, adjusting the parameters of the first annotation quality assessment model by using the gradient descent method;

[0043] The stability of the model parameter adjustment process is improved through the adaptive learning rate adjustment mechanism.

[0044] Furthermore, the stability of the model parameter adjustment process is improved through the adaptive learning rate adjustment mechanism, including:

[0045] Adjust the learning rate in real time according to the rate of gradient descent during each training process;

[0046] Set the initial learning rate , the learning rate adjustment formula is:

[0047] ,

[0048] in, For the The learning rate during the training iteration, is the initial learning rate, is a fixed learning rate decay factor, No. Dynamic adjustment factor during training iteration, is the current iteration training number, For the The gradient value during the training iteration is , is a constant;

[0049] Dynamic Adjustment Factor The adjustment formula is:

[0050] ,

[0051] in, For the Dynamic adjustment factor during training iteration, The loss function is The rate of change in training iterations, , and Respectively Second and The training difference of training iterations, For the The gradient change rate in the training iteration, , and Respectively Second and The gradient value of the training iteration, is a constant.

[0052] Furthermore, unlabeled task data is obtained, and the task data is assigned to annotators according to the data that the annotators are good at. The annotators annotate the data to obtain a real-time labeled data set, including:

[0053] According to the data that the annotator is good at, the matching score of the annotator is obtained. The calculation formula of the matching score is:

[0054] ,

[0055] in, For the annotator For The matching scores of the types of data are For the annotator In the The average historical annotation accuracy of the data types, For the annotator In the The number of historical annotations of each type of data, For the annotator For The labeling efficiency of different types of data, is the normalization coefficient, is the coefficient;

[0056] According to the matching scores of the annotators, the task data is assigned to the annotators in the following way:

[0057] ,

[0058] in, For the Whether the task data of this type is assigned to the annotator As a result, For the annotator For The matching scores of the data types are: For the A marker, is the total number of annotators;

[0059] When the annotator is assigned task data, he annotates the task data to obtain a real-time annotated data set.

[0060] Furthermore, the average historical annotation accuracy The calculation formula is:

[0061] ,

[0062] in, For the annotator In the The average historical annotation accuracy of the data types, For the annotator In the The number of historical annotations for each type of data, For the annotator In the Type of data The historical annotation accuracy of historically annotated data;

[0063] Labeling efficiency The calculation formula is:

[0064] ,

[0065] in, For the annotator For The labeling efficiency of different types of data, For the annotator In the The average historical annotation accuracy of the data types, For the annotator Complete The average time of the data type, and is the coefficient.

[0066] Furthermore, the calculation formula for the consistency of the annotation data is:

[0067] ,

[0068] in, For the Type task data The consistency of the data annotation results, For the The number of type annotation task data, For the The number of annotators who annotate the task data of type, For the The weight of the annotator, is the similarity function, , For the The annotator The type annotation task The labeling results given by the data For the The annotator The type annotation task The labeling results given by the data, and , As parameters, To adjust the function, , For the Parameters of the labeled data, as a parameter.

[0069] The above solution of the present invention includes at least the following beneficial effects:

[0070] The present invention screens the annotator's historical annotation data and combines it with historical usage scores to extract high-quality annotation data as the basis for building an evaluation model. This data screening method avoids the interference of low-quality data on the model and ensures that the model has high accuracy in the initial stage. By acquiring the annotation data in real time and inputting it into the evaluation model for dynamic scoring, the model can quickly respond to changes in the annotator's ability, improve the accuracy of the evaluation, and avoid the limitations of the traditional single scoring method.

[0071] The present invention analyzes the annotators' historical annotation quality and areas of expertise, determines the type of data that each annotator is best at processing, and assigns corresponding annotation tasks based on the results. This method not only optimizes the accuracy of task assignment, but also improves the overall efficiency of annotation. Through reasonable task assignment, annotators can give full play to their strengths, thereby improving the accuracy and consistency of data annotation and avoiding the impact of tasks that they are not good at on the annotation results.

[0072] The present invention evaluates the labeled data in real time. The evaluation results are not only used to judge the quality of the current labeled data, but also can optimize the labeling quality evaluation model through the self-adjustment mechanism of the model; by comparing the first quality score and the second quality score in the real-time labeled data set, calculating the deviation value, and then making preliminary adjustments to the evaluation model according to the deviation value, this dynamic optimization method ensures that the model can be quickly adjusted as the ability of the annotator changes, so that it can always maintain efficient and accurate evaluation capabilities.

[0073] By introducing a real-time scoring mechanism, the present invention can quickly obtain the latest annotation performance of the annotator, thereby adjusting the allocation of annotation tasks in a timely manner and improving the response speed of the model to changes in annotation quality. In traditional methods, the use of scoring feedback for annotation data usually takes a long time, resulting in a lag in the assessment of annotation quality. By continuously receiving and analyzing real-time annotation data, the model can make adjustments in a short time, reducing the lag in annotation quality assessment.

[0074] The present invention does not rely on a single scoring standard by comparing the first quality score with the second quality score. The first quality score is a real-time score of the labeled data itself, while the second quality score is a score based on the consistency between the labeled data. By comparing the scores in two different dimensions, the quality of the labeled data can be judged more comprehensively, thereby reducing the errors that may be caused by a single score. This multi-angle evaluation method makes the model more reliable.

[0075] The present invention receives usage score feedback of real-time annotated data and adjusts the evaluation model secondary to enable it to gradually adapt to new data characteristics. This adaptive learning method enables the evaluation model to maintain good performance for a long time, especially in the face of changes in the annotator's ability or increased data complexity, it can still maintain high accuracy and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 A schematic diagram of a method for evaluating the quality of labeled data based on active learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0077] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0078] like Figure 1 As shown, an embodiment of the present invention proposes a method for evaluating the quality of labeled data based on active learning, and the method includes:

[0079] Step 11, obtaining the annotator's historical annotation data and historical usage scores, and screening the historical annotation data according to the historical usage scores to obtain high-quality annotation data and its high-quality usage scores;

[0080] Step 12: construct an annotation quality assessment model based on the high-quality annotation data and the high-quality usage score;

[0081] Step 13, analyzing the type of annotation data that the annotator is good at based on the high-quality usage score, and obtaining the data that the annotator is good at;

[0082] Step 14, obtaining unlabeled task data, and assigning the task data to annotators based on the data that the annotators are good at, and the annotators annotate it to obtain a real-time labeled data set;

[0083] Step 15, inputting the real-time annotation data set into the annotation quality assessment model to obtain a first quality score;

[0084] Step 16, interactively verifying the annotated data in the real-time annotated data set, calculating the consistency of the annotated data, and obtaining a second quality score;

[0085] Step 17, comparing the first quality score with the second quality score to obtain a deviation value;

[0086] Step 18: when the deviation value is less than a preset threshold, the annotation quality assessment model is preliminarily adjusted according to the real-time annotation data set;

[0087] Step 19, after receiving the usage score of the real-time annotation data set, the annotation quality assessment model is adjusted again by using the usage score.

[0088] In an embodiment of the present invention, by obtaining the historical annotation data and usage scores of the annotators, high-quality annotation data is screened out, and then a annotation quality assessment model is constructed. Compared with the existing technology, this method not only takes into account the historical performance of the annotators, but also further improves the scientific nature of data screening based on the usage scores, thereby ensuring the high quality of the initial data of the model. The key to this process is to select high-quality annotation data through historical data as the basis for model training, which can effectively reduce the interference of low-quality data on model construction and improve the initial accuracy of the annotation quality assessment model; analyze the type of data that the annotator is good at processing, and reasonably allocate the unlabeled task data to the appropriate annotator. This task allocation mechanism improves the efficiency and accuracy of annotation. By allocating data that meets the type of expertise of the annotator, the ability of the annotator can be fully utilized, avoiding the problem of decreased data annotation quality caused by incorrect allocation.

[0089] By inputting the annotator's real-time annotated data set into the annotation quality assessment model, obtaining the first quality score and calculating the consistency of the annotation data, the annotation quality can be evaluated in multiple dimensions; by comparing the first quality score with the second quality score, the deviation value is calculated, and when the deviation value is less than the preset threshold, the assessment model is adjusted to ensure that the assessment model is always in the best state. This method is that the assessment model can be dynamically adjusted to adapt to the fluctuation of the annotator's ability, avoiding the problem of lagging model updates in the prior art, and after receiving the usage score of the real-time annotated data set, a secondary adjustment is made through the usage score to further optimize the model, ensuring that the model has the ability to continuously learn and optimize during long-term use.

[0090] Among them, step 11 collects all types of annotated data completed and submitted to the customer by the annotator, including different types of annotation results such as text, image, audio or video, and obtains the usage score of these historical annotated data. The usage score comes from customer feedback and reflects the quality and accuracy of the annotated data. After obtaining enough historical annotated data and its usage score, the historical annotated data is screened by the usage score of the historical annotated data. The screening can be based on setting a score threshold. For example, the full score is 100, and the threshold is set to 85. Only the annotated data exceeding this threshold is considered to be high-quality. Through the above screening process, a group of high-quality annotated data can be identified and extracted from the historical annotated data. At the same time, the high-quality usage scores corresponding to these high-quality annotated data are also retained. These high-quality usage scores can not only be used to further verify the reliability of the data, but also can be used as a quantitative indicator of the annotator's ability.

[0091] Among them, step 12, based on the long short-term memory network model, configures the input layer, feature extraction layer, quality assessment layer and output layer, constructs a first annotation quality assessment model, and divides the high-quality annotation data into a training set and a test set, inputs the training set into the first annotation quality assessment model, obtains a training quality score (predicted value), and compares it with the high-quality usage score (true value) of the high-quality annotation data to obtain a training difference, and optimizes the first annotation quality assessment model through the training difference to obtain a preliminary annotation quality assessment model, and tests the preliminary annotation quality assessment model through the test set to verify the quality assessment accuracy of the preliminary annotation quality assessment model, thereby obtaining a annotation quality assessment model.

[0092] Among them, in step 13, the high-quality usage score of the high-quality annotated data represents the excellent annotation performance of the annotator in the annotation task. In order to analyze the type of annotation data that the annotator is good at, it is necessary to classify the usage score according to different types of annotation data. First, all high-quality annotated data are classified according to their types. For example, text type data, image type data, and audio type data are classified into different categories. After the classification is completed, the high-quality usage score of each type of data is statistically analyzed. Specifically, all the high-quality usage scores in each type of annotation data are added and divided by the total number of annotation data of this type to obtain the average usage score. This average usage score can be used as a quantitative indicator of the overall performance of the annotator on this type of data. After obtaining the average usage score of different types of data, the relative performance of the annotator in various tasks is further analyzed. A threshold can be set to filter out the task type that the annotator is best at. For example, a threshold higher than the overall average level of the annotators is set. Only when the average usage score of the annotator on a certain type of data exceeds this threshold, the data of this type will be considered as the type that the annotator is good at.

[0093] Among them, step 14, according to the type that the annotator is good at, the matching score calculation formula is used to calculate the matching score of the type of the annotator, and according to the numerical value of the matching score of the annotator, the task data of the same type is assigned to the annotator with a high matching score. After the assignment is completed, the annotator annotates the assigned task data to obtain real-time annotation data.

[0094] Among them, in step 15, the real-time annotation data set is input into the annotation quality assessment model, and the annotation quality assessment model outputs the quality score of the real-time annotation data, thereby obtaining a first quality score.

[0095] Among them, step 16 is to interactively verify the annotated data in the real-time annotated data set, and calculate the consistency of the annotated data through a consistency calculation formula, so as to obtain a second quality score, which represents the consensus of the annotators on the task data.

[0096] Among them, in step 17, the first quality score is subtracted from the second quality score and the absolute value is taken to obtain a deviation value. The size of the deviation value represents the annotation quality of the real-time data set to a certain extent.

[0097] Among them, in step 18, when the deviation value is less than the preset threshold, the annotation quality assessment model is preliminarily adjusted according to the real-time annotation data set; the deviation value is compared with the preset threshold, and when the deviation value is less than the preset threshold, it means that the annotation quality of the real-time annotation data set is high-quality, then the real-time annotation data set is converted into a vector format, and input into the annotation quality assessment model, and the annotation quality assessment model is adjusted, and the parameters of the annotation quality assessment model are adjusted until the rate of change of the deviation value decreases is lower than the preset rate of change threshold; if the deviation value is not less than the preset threshold, it means that the first quality score and the second quality score are too different, and the annotation result of the real-time annotation data set is uncertain, and it is necessary to wait for the customer's usage score feedback for judgment.

[0098] Among them, step 19, after receiving the usage score of the real-time annotation data set, the annotation quality assessment model is adjusted twice by using the usage score; when the customer's usage score feedback on the real-time annotation data set is obtained, the pros and cons of the usage score are judged according to the preset threshold. If the usage score is excellent, the real-time annotation data set is converted into a vector format and input into the annotation quality assessment model, and the annotation quality assessment model is adjusted twice to adjust the parameters of the annotation quality assessment model until the difference between the first quality score and the real-time annotation data usage score is less than the preset threshold, and the decreasing rate of change of the difference is lower than the preset change rate threshold, then the secondary adjustment process is stopped; if the usage score is poor, the task data is reallocated and labeled.

[0099] In a preferred embodiment of the present invention, step 12 comprises:

[0100] Preprocess the high-quality labeled data to obtain standardized data, and divide the standardized data into training data set and test data set through stratified sampling;

[0101] Based on the long short-term memory network model, the first annotation quality assessment model is constructed;

[0102] Inputting the training data set into the first annotation quality assessment model to obtain a training quality score;

[0103] The difference between the training quality score and the high-quality usage score is calculated through the loss function to obtain the training difference;

[0104] According to the training difference, adjusting the parameters of the first annotation quality assessment model;

[0105] When the training difference is lower than the preset threshold and the rate of change of its downward trend meets the preset rate of change, the training is stopped to obtain a preliminary labeling quality assessment model;

[0106] Input the test data set into the preliminary annotation quality assessment model to obtain the test quality score;

[0107] The difference between the test quality score and the high-quality usage score is calculated through the loss function to obtain the test difference;

[0108] When the test difference is within the preset threshold, the annotation quality assessment model is obtained. When the test difference is not within the preset threshold, retraining is performed and the direction of the adjustment parameters is modified.

[0109] In an embodiment of the present invention, the labeled data is first preprocessed to obtain standardized data, and the data is divided into a training set and a test set by stratified sampling to ensure the diversity and balance of the data set. This process lays a solid foundation for the training and testing of subsequent models. Then, a first labeling quality assessment model is constructed based on the long short-term memory network model. The long short-term memory network model can effectively capture the long-term dependencies between the labeled data, which helps to improve the performance of the labeling quality assessment model in complex data. By inputting the training data set, the labeling quality assessment model generates a training quality score, and uses the loss function to calculate the difference between the training quality score and the quality use score, so as to adjust the parameters of the labeling quality assessment model according to this difference until the training error of the labeling quality assessment model meets the preset standard. By introducing the long short-term memory network model, complex labeled data can be better processed, especially for data types with time series dependence, and higher prediction accuracy can be provided. In addition, through the optimization of the loss function, the risk of overfitting or underfitting in the model training process is effectively reduced, thereby improving the generalization ability of the model.

[0110] Among them, the high-quality labeled data is preprocessed to obtain standardized data, and the standardized data is divided into a training data set and a test data set through stratified sampling. Specifically, the high-quality labeled data is preprocessed to obtain standardized data, and the preprocessing includes removing noise data, filling missing data, unifying data formats, and standardizing or normalizing data of specific numerical types; then the standardized data is divided into a training data set and a test data set through stratified sampling. First, the standardized data is classified by category, and the proportion of each type of data in the entire data set is calculated. Then, the different types of data in the training data set and the test data set also need to maintain this ratio, and the standardized data is divided into a training set and a test set according to a preset ratio of 80 / 20. Then, for each type of data, data sampling is performed according to the preset division ratio to ensure that each type of data is distributed in the same proportion. During the sampling process, sampling is performed randomly, and after stratified sampling, a training data set and a test data set are generated respectively.

[0111] In a preferred embodiment of the present invention, it is characterized in that a first annotation quality assessment model is constructed based on a long short-term memory network model, including:

[0112] Configure the input layer to receive the input labeled data and obtain the input data;

[0113] Configure the feature extraction layer to extract feature vectors of input data and high-quality labeled data to obtain a vector set;

[0114] Configure a quality assessment layer to process the vector set and generate a quality score for the input data;

[0115] Configure the output layer to output the quality score of the input data;

[0116] A first annotation quality assessment model is constructed according to the input layer, the feature extraction layer, the quality assessment layer and the output layer.

[0117] In an embodiment of the present invention, the structure of the first annotation quality assessment model is described in detail. The annotation quality assessment model is configured with an input layer, a feature extraction layer, a quality assessment layer, and an output layer to ensure that each step from inputting annotation data to generating a score can be processed efficiently and accurately. The design of the feature extraction layer enables the first annotation quality assessment model to automatically extract key features from the annotation data, while the quality assessment layer is responsible for generating the final quality score based on these features; compared with traditional annotation quality assessment methods, the first annotation quality assessment model has a strong automatic feature extraction capability, which can greatly reduce the necessity of human intervention, thereby improving the automation level and accuracy of the assessment process.

[0118] Among them, the feature vectors of input data and high-quality annotated data are extracted. The first annotation quality assessment model can learn and capture the key features in the data, thereby providing a basis for subsequent quality assessment; the annotation quality assessment model automatically extracts the key features representing the data from it. In this process, the following operations are involved:

[0119] For image data, the first annotation quality assessment model uses convolution operations to extract spatial features in the image;

[0120] For text data or sequence data, the first annotation quality assessment model maps discrete input data into a continuous vector space to generate a vector representation of the vocabulary or sequence;

[0121] For structured data, the first annotation quality assessment model extracts statistical features such as mean, variance, and extreme values, or extracts more representative features based on the type of data, such as trends and periodicity in time series.

[0122] Among them, the first annotation quality assessment model is constructed according to the input layer, feature extraction layer, quality assessment layer and output layer. The configured input layer, feature extraction layer, quality assessment layer and output layer can be connected in this order. During the connection process, it is necessary to ensure that the output of each layer can be correctly passed to the next input, thereby completing the structural composition of the first annotation quality assessment model and constructing the first annotation quality assessment model.

[0123] In a preferred embodiment of the present invention, the vector set is processed to generate a quality score of the input data, including:

[0124] Perform linear transformation on the vector set and calculate the intermediate vector;

[0125] The intermediate vector is activated by a nonlinear activation function to obtain an activation vector;

[0126] Mapping activation vectors to usage scores of good labeled data produces quality scores for the input data.

[0127] In an embodiment of the present invention, a process of processing the features of the annotation data is described, the feature vectors of the input data and the high-quality annotation data are merged, and a final quality score is generated through steps such as linear transformation and nonlinear activation function; by introducing the nonlinear activation function, the first annotation quality assessment model can better handle complex feature relationships, making the assessment results more accurate, and the application of the nonlinear activation function improves the performance of the first annotation quality assessment model when processing complex data, so that the relationship between features can be better captured and characterized, ultimately improving the accuracy and reliability of the annotation quality assessment.

[0128] Among them, a linear transformation is performed on the vector set to calculate the intermediate vector. Specifically, the linear transformation refers to linearly combining the input feature vectors through matrix operations to generate new feature vectors, that is, the weight matrix and the bias vector are initialized according to the parameter settings of the first annotation quality assessment model, and the weight matrix and the bias vector are generated in a normal distribution manner. Then, the vector set and the weight matrix are matrix multiplied. After the matrix multiplication is completed, the bias vector is added to the calculation result, that is, each element in the bias vector is added to the corresponding element in the transformed result vector. After the multiplication of the weight matrix and the addition of the bias vector, the system generates a new representation vector, namely the intermediate vector.

[0129] Among them, the intermediate vector is activated by a nonlinear activation function to obtain an activation vector. Specifically, the intermediate vector is activated by a Sigmoid function, and the expression of the activation function is:

[0130] ,

[0131] in, is the output value after activation, is any component of the intermediate vector.

[0132] Through nonlinear activation function processing, the first annotation quality assessment model can extract more complex features in the input data and capture the nonlinear relationships in the input features. Nonlinear activation helps the annotation quality assessment model cope with these complexities and generate more accurate quality scores.

[0133] Among them, the activation vector is mapped to the usage score of the high-quality labeled data to generate the quality score of the input data. Specifically, the mapping process is that the first labeling quality assessment model processes the activation vector once to generate the quality score of the input data. In this embodiment, the labeling quality assessment model establishes a mapping relationship between the activation vector and the usage score of the high-quality labeled data. The mapping relationship is expressed as a one-to-one correspondence between the input data and the high-quality labeled data in the activation vector. Through this correspondence, the quality score of the input data and the usage score relationship of the high-quality labeled data can be mapped. The mapped result is the quality score of the input data, which represents the evaluation result of the labeling quality assessment model on the input label data.

[0134] In a preferred embodiment of the present invention, the calculation formula of the loss function is:

[0135] ,

[0136] in, is the loss function, The training data set The training quality score of the labeled data, The training data set The usage score of the high-quality labeled data corresponding to the labeled data, is the total number of labeled data in the training dataset, is the regularization coefficient, is the L2 regularization term of the first labeled data quality assessment model parameters, The training data set The variance of the quality score of the labeled data, is the coefficient.

[0137] In an embodiment of the present invention, a method for calculating the loss function is described in detail. The loss function not only considers the difference between the quality score of the input data and the score of the high-quality labeled data, but also introduces a regularization term and the variance of the quality score to better adjust the model parameters and prevent the occurrence of overfitting. Through the optimized loss function formula, the model can simultaneously consider the accuracy and stability of the data during calculation, ensuring that high-quality evaluation results can be maintained when facing different data sets. Compared with existing model training methods, the addition of the variance term enables the model to not only focus on the accuracy of the labeled data, but also better cope with changes in scores under different data complexities.

[0138] Among them, the loss function is a function of the difference between the predicted results of the annotation quality assessment model and the actual results. In this embodiment, the loss function can measure the error between the quality score predicted by the annotation quality assessment model and the usage score of the high-quality annotation data. The smaller the value of the loss function, the closer the prediction of the annotation quality assessment model is to the actual value.

[0139] Among them, the variance term It can measure the variance of the input data quality score. This item penalizes the fluctuation of the quality score to prevent the annotation quality assessment model from being too dependent on certain high-variance data points; the coefficient Control the effect of the variance penalty term on the loss function by adjusting The value of can adjust the sensitivity of the annotation quality assessment model to data variance. If the variance term accounts for too large a proportion, the annotation quality assessment model may be over-smoothed, while if the proportion is too small, the annotation quality assessment model may be too dependent on high variance data.

[0140] Among them, regularization can prevent the model from overfitting. Regularization limits the size of the annotation quality assessment model parameters by adding a penalty term to avoid the annotation quality assessment model from overfitting the training data, thereby improving the generalization ability of the annotation quality assessment model. In this embodiment, L2 regularization is performed by The square weighted penalty is used to make the value of the annotation quality assessment model parameter as small as possible to ensure that the annotation quality assessment model is more robust. Excessive parameters may lead to overfitting of the model, and the regularization term limits the occurrence of this situation by penalizing excessive parameters.

[0141] In a preferred embodiment of the present invention, it is characterized in that adjusting the parameters of the first annotation quality assessment model according to the training difference includes:

[0142] According to the size and direction of the training difference, the gradient of the loss function relative to the first labeled data quality assessment model parameters is obtained;

[0143] The gradients of all parameters in the first annotation quality assessment model are obtained by the chain rule to generate gradient change information;

[0144] According to the gradient change information, adjusting the parameters of the first annotation quality assessment model by using the gradient descent method;

[0145] The stability of the model parameter adjustment process is improved through the adaptive learning rate adjustment mechanism.

[0146] In an embodiment of the present invention, the parameter adjustment of the annotation quality assessment model is performed by the gradient descent method. By calculating the gradient of the training difference relative to the model parameters, the current learning direction of the annotation quality assessment model can be determined, and the parameters can be adjusted accordingly. The annotation quality assessment model calculates the gradients of all parameters in the annotation quality assessment model by the chain rule, thereby generating accurate gradient change information. Compared with the prior art, the combination of the gradient descent method and the adaptive learning rate adjustment mechanism significantly improves the stability and convergence speed of the annotation quality assessment model during the training process. The adaptive learning rate enables the annotation quality assessment model to dynamically adjust the learning rate according to the gradient descent rate, thereby avoiding the training instability problem that may be caused by the fixed learning rate in the traditional gradient descent method. This method not only speeds up the training speed of the annotation quality assessment model, but also improves the training accuracy and reduces the uncertainty in the tuning process of the annotation quality assessment model.

[0147] Among them, according to the size and direction of the training difference, the gradient of the loss function relative to the parameters of the first labeled data quality assessment model is obtained. Specifically, the gradient is the partial derivative of the loss function relative to the parameters of the labeled quality assessment model, that is, the gradient of the parameter is obtained by calculating the partial derivative of the loss function with respect to the parameter; the size and direction of the training difference determines the size of the gradient value. If the training difference is large, the gradient value will also be relatively large, and the model will make larger adjustments to the parameters to quickly reduce the error. If the training difference is small, the gradient value is small, and the model parameters will be fine-tuned to a smaller extent. The size and direction of the training difference determine the update direction and strength of the gradient to ensure that the model can gradually approach the optimal state.

[0148] Among them, the gradients of all parameters in the first annotation quality assessment model are obtained through the chain rule to generate gradient change information. Specifically, in the first annotation quality assessment model, the loss function is a nested combination of multiple functions. The annotation quality assessment model calculates the influence of each parameter on the loss function layer by layer through the chain rule, thereby obtaining the overall gradient of the first annotation quality assessment model and generating gradient change information.

[0149] Among them, according to the gradient change information, the parameters of the first annotation quality assessment model are adjusted by the gradient descent method. Specifically, the gradient descent method can gradually reduce the loss function by continuously adjusting the model parameters, and update the parameters along the direction of gradient descent, that is, according to the derivative information of the loss function with respect to the parameters, the model parameters are updated with a certain step size so that the loss function gradually tends to the minimum, and the process of calculating the gradient and updating the parameters is repeated until the value of the loss function reaches a preset threshold.

[0150] In a preferred embodiment of the present invention, the stability of the model parameter adjustment process is improved through an adaptive learning rate adjustment mechanism, including:

[0151] Adjust the learning rate in real time according to the rate of gradient descent during each training process;

[0152] Set the initial learning rate , the learning rate adjustment formula is:

[0153] ,

[0154] in, For the The learning rate during the training iteration, is the initial learning rate, is a fixed learning rate decay factor, No. Dynamic adjustment factor during training iteration, is the current iteration training number, For the The gradient value during the training iteration is , is a constant;

[0155] Dynamic Adjustment Factor The adjustment formula is:

[0156] ,

[0157] in, For the Dynamic adjustment factor during training iteration, The loss function is The rate of change in training iterations, , and Respectively Second and The training difference of training iterations, For the The gradient change rate in the training iteration, , and Respectively Second and The gradient value of the training iteration, is a constant.

[0158] In an embodiment of the present invention, the adjustment formula of the adaptive learning rate is updated in real time according to the gradient changes in each training process, so that the model can converge faster in the early stage of training, and fine-tune with a smaller step size in the later stage of training to ensure the stability and convergence effect of the model; this mechanism is particularly prominent in solving the training of large-scale data sets. By dynamically adjusting the learning rate, the problems of too fast convergence or too slow learning speed that may be caused by the traditional fixed learning rate scheme are effectively avoided. Compared with the existing learning rate adjustment method, the dynamic mechanism of this embodiment improves the flexibility and efficiency of model training, so that it can maintain a high performance when facing complex labeling tasks.

[0159] Among them, the learning rate is an important hyperparameter in the gradient descent method, which controls the step size each time the model parameters are updated. If the learning rate is too large, the model may skip the optimal solution, resulting in an unstable or even divergent training process. If the learning rate is too small, the training process may be too slow and the optimal solution cannot be found quickly.

[0160] Among them, the fixed learning rate decay factor Control the basic decay of the learning rate so that the learning rate gradually decreases during the training process. As the learning rate increases, The rate of gradually decreases to ensure that the annotation quality assessment model can transition from the initial rapid training to the later fine-tuning; when When it is small, the decay is slow and the learning rate is relatively large, which helps to quickly optimize the annotation quality assessment model. When it is larger, the decay is faster and the learning rate gradually decreases, avoiding oscillation of the labeling quality assessment model when it is close to the optimal solution.

[0161] The dynamic adjustment factor It is related, indicating that the learning rate is adjusted according to the gradient change of the annotation quality assessment model in each iteration. Through the gradient sum of squares, the formula can dynamically adjust the learning rate according to the size of the gradient; the cumulative value of the gradient sum of squares reflects the changing trend of the annotation quality assessment model during the training process. If the gradient changes greatly, it means that the parameters of the annotation quality assessment model need to be adjusted greatly. At this time, the learning rate can be increased accordingly to speed up the training. If the gradient changes little, it means that the annotation quality assessment model is close to the optimal solution. At this time, the learning rate is reduced to avoid excessive parameter updates. This mechanism helps to improve the adaptability of the annotation quality assessment model training, especially when processing data of different complexities.

[0162] in, is a constant with a very small value, which is used to avoid the situation where the denominator is zero.

[0163] In a preferred embodiment of the present invention, unlabeled task data is obtained, and the task data is assigned to annotators according to the data that the annotators are good at, and the annotators annotate the task data to obtain a real-time labeled data set, including:

[0164] According to the data that the annotator is good at, the matching score of the annotator is obtained. The calculation formula of the matching score is:

[0165] ,

[0166] in, For the annotator For The matching scores of the data types are: For the annotator In the The average historical annotation accuracy of the data types, For the annotator In the The number of historical annotations of each type of data, For the annotator For The labeling efficiency of different types of data, is the normalization coefficient, is the coefficient;

[0167] According to the matching scores of the annotators, the task data is assigned to the annotators in the following way:

[0168] ,

[0169] in, For the Whether the task data of this type is assigned to the annotator As a result, For the annotator For The matching scores of the data types are: For the A marker, is the total number of annotators;

[0170] When the annotator is assigned task data, he annotates the task data to obtain a real-time annotated data set.

[0171] In an embodiment of the present invention, the allocation of task data is based on the matching score of the annotator. The rationality of task allocation is optimized by the matching score, which ensures the best match between the task and the annotator's ability, thereby improving the overall efficiency of the annotation task and the quality of the annotation data. The calculation of the matching score fully considers the performance of the annotator in the historical annotation data, including factors such as the average historical annotation accuracy, the number of historical annotations, and the annotation efficiency. Through the comprehensive evaluation of these indicators, it can ensure that the allocation of task data is more reasonable, and effectively improve the completion efficiency of the annotation task and the quality of the annotation data.

[0172] Among them, the calculation formula of the matching score takes into account the characteristics of the annotator in multiple dimensions, ensuring that the annotator's ability and task requirements can be reasonably matched during the task allocation process, avoiding the problem of reduced annotation quality due to mismatch of task types.

[0173] in, For the annotator In the The historical annotation times of each type of data are directly obtained from the historical annotation data.

[0174] In a preferred embodiment of the present invention, the average historical annotation accuracy The calculation formula is:

[0175] ,

[0176] in, For the annotator In the The average historical annotation accuracy of the data types, For the annotator In the The number of historical annotations for each type of data, For the annotator Type of data The historical annotation accuracy of historically annotated data;

[0177] Labeling efficiency The calculation formula is:

[0178] ,

[0179] in, For the annotator For The labeling efficiency of different types of data, For the annotator In the The average historical annotation accuracy of the data types, For the annotator Complete The average time of the data type, and is the coefficient.

[0180] In an embodiment of the present invention, the calculation of the average historical annotation accuracy and annotation efficiency of the annotator in different types of data helps to improve the accuracy of task assignment. By analyzing the historical annotation results, the performance of the annotator in different types of tasks can be accurately evaluated, thereby providing a basis for task assignment. The matching score calculation formula not only takes into account the accuracy of the annotator, but also combines factors such as annotation efficiency and annotation task completion time, so that both annotation quality and efficiency can be taken into account when assigning tasks, ensuring that the tasks can be completed with high quality within the specified time.

[0181] in, For the annotator In the The average historical annotation accuracy of a type of data indicates, to a certain extent, the annotation quality pass rate of the annotator for this type of data.

[0182] in, For the annotator Complete The average time for each type of data is obtained from the annotator's historical annotation data.

[0183] in, For the annotator For The labeling efficiency of the data type reflects the labeler’s The comprehensive ability in various types of tasks, annotators with high comprehensive ability will be assigned high-difficulty tasks first.

[0184] In a preferred embodiment of the present invention, the calculation formula for the consistency of the annotation data is:

[0185] ,

[0186] in, For the Type task data The consistency of the data annotation results, For the The number of type annotation task data, For the The number of annotators who annotate the task data of type, For the The weight of the annotator, is the similarity function, , For the The annotator The type annotation task The labeling results given by the data For the The annotator The type annotation task The labeling results given by the data, and , As parameters, To adjust the function, , For the Parameters of the labeled data, as a parameter.

[0187] In an embodiment of the present invention, the annotation results of different annotators on the same task are compared by interactive verification, the similarity between the annotation data is calculated, and a second quality score is generated based on the similarity. The quality score calculation formula takes into account the weights of different annotators and the complexity of the task data, so that the score can better reflect the actual quality of the annotation data, thereby providing a reliable basis for subsequent annotation data quality control.

[0188] in, For other The weight of each annotator. When different annotators annotate the same type of data, their contributions to the overall consistency will be different due to their different experiences, abilities, performances, etc. Different weights are given to each annotator to balance the influence of different annotators; the annotation results of some annotators may be more reliable, so their annotations will contribute more to the consistency assessment, while for annotators with poor performance, their weights may be smaller, thereby reducing their negative impact on the consistency results.

[0189] Among them, the complexity parameter To measure the The complexity of a labeling task has a direct impact on the labeling results. The difficulty parameter usually takes into account the complexity of the unlabeled data, the professional requirements of the task, and the ambiguity of the labeling task itself. Labeled data sometimes increases the difficulty of labeling due to ambiguity, ambiguity, or complexity in details. For example, when classifying images, some images may contain multiple objects, or the boundaries of different objects are unclear, making labeling difficult. For such complex labeling tasks, the complexity parameter will be higher to reflect the difficulty of the annotation process.

[0190] Among them, the adjustment function The consistency evaluation is adjusted by combining the complexity of the labeled data and the individual characteristics of the annotator to ensure that the evaluation process can be reasonably adjusted according to the annotator's ability and the complexity of the annotation. The core idea is to match the annotator's characteristics and ability with the complexity of the labeled data to make the evaluation more fair and flexible. The performance of different annotators on the same task will vary due to their experience, professional background, and areas of expertise. The role of is to adjust the task complexity according to the individual characteristics of the annotator. For example, if an annotator has always performed well on a specific type of task, their annotation consistency is expected to remain high even when faced with more complex tasks. Conversely, annotators with weaker performance may show large differences even when faced with simpler tasks.

[0191] Among them, the similarity function It is mainly used to measure the consistency of different annotators on the same annotation task. In the annotation task, multiple annotators annotate the same data, and the difference in annotation results may come from the complexity of the annotators or the data itself. Through the similarity function, the results of different annotators can be quantitatively compared and their consistency can be evaluated; this similarity function can convert Euclidean distance into similarity value, and the value range is between [0,1].

[0192] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. The labeling data quality assessment method based on active learning is characterized by: The method comprises: Obtain the annotator's historical annotation data and historical usage scores, and filter the historical annotation data according to the historical usage scores to obtain high-quality annotation data and its high-quality usage scores; Build an annotation quality assessment model based on high-quality annotation data and high-quality usage scores; According to the high-quality usage score, analyze the type of annotation data that the annotator is good at, and obtain the data that the annotator is good at; Obtain unlabeled task data, and assign the task data to annotators based on their expertise. The annotators annotate the data to obtain a real-time labeled data set. Inputting the real-time annotation data set into the annotation quality assessment model to obtain a first quality score; Interactively verify the annotated data in the real-time annotated data set, calculate the consistency of the annotated data, and obtain a second quality score; By comparing the first quality score with the second quality score, a deviation value is obtained; When the deviation value is less than the preset threshold, the annotation quality assessment model is preliminarily adjusted according to the real-time annotation data set; After receiving the usage score of the real-time annotation dataset, the annotation quality assessment model is adjusted again by using the usage score; Based on high-quality annotation data and high-quality usage scores, we build an annotation quality assessment model, including: Preprocess the high-quality labeled data to obtain standardized data, and divide the standardized data into training data set and test data set through stratified sampling; Based on the long short-term memory network model, the first annotation quality assessment model is constructed; Inputting the training data set into the first annotation quality assessment model to obtain a training quality score; The difference between the training quality score and the high-quality usage score is calculated through the loss function to obtain the training difference; According to the training difference, adjusting the parameters of the first annotation quality assessment model; When the training difference is lower than the preset threshold and the rate of change of its downward trend meets the preset rate of change, the training is stopped to obtain a preliminary labeling quality assessment model; Input the test data set into the preliminary annotation quality assessment model to obtain the test quality score; The difference between the test quality score and the high-quality usage score is calculated through the loss function to obtain the test difference; When the test difference is within the preset threshold, the annotation quality assessment model is obtained. When the test difference is not within the preset threshold, retraining is performed and the direction of the adjustment parameters is modified. Based on the long short-term memory network model, the first annotation quality assessment model is constructed, including: Configure the input layer to receive the input labeled data, obtain the input data, and obtain the input data; Configure the feature extraction layer to extract feature vectors of input data and high-quality labeled data to obtain a vector set; Configure a quality assessment layer to process the vector set and generate a quality score for the input data; Configure the output layer to output the quality score of the input data; Constructing a first annotation quality assessment model according to the input layer, the feature extraction layer, the quality assessment layer and the output layer; Process the vector set to generate a quality score for the input data, including: Perform linear transformation on the vector set and calculate the intermediate vector; The intermediate vector is activated by a nonlinear activation function to obtain an activation vector; Mapping activation vectors to usage scores of high-quality annotated data to generate quality scores for input data; The loss function is calculated as: ， in, is the loss function, The training data set The training quality score of the labeled data, The training data set The usage score of the high-quality labeled data corresponding to the labeled data, is the number of labeled data in the training dataset, is the regularization coefficient, is the L2 regularization term of the first labeled data quality assessment model parameters, The training data set The variance of the quality score of the labeled data, is the coefficient.

2. The method for evaluating the quality of labeled data based on active learning according to claim 1, characterized in that: According to the training difference, the parameters of the first annotation quality assessment model are adjusted, including: According to the size and direction of the training difference, the gradient of the loss function relative to the first labeled data quality assessment model parameters is obtained; The gradients of all parameters in the first annotation quality assessment model are obtained by the chain rule to generate gradient change information; According to the gradient change information, adjusting the parameters of the first annotation quality assessment model by using the gradient descent method; The stability of the model parameter adjustment process is improved through the adaptive learning rate adjustment mechanism.

3. The method for evaluating the quality of labeled data based on active learning according to claim 1, characterized in that: The adaptive learning rate adjustment mechanism improves the stability of the model parameter adjustment process, including: Adjust the learning rate in real time according to the rate of gradient descent during each training process; Set the initial learning rate , the learning rate adjustment formula is: ， in, For the The learning rate during the training iteration, is the initial learning rate, is a fixed learning rate decay factor, No. Dynamic adjustment factor during training iteration, is the current iteration training number, For the The gradient value during the training iteration is , is a constant; Dynamic Adjustment Factor The adjustment formula is: ， in, For the Dynamic adjustment factor during training iteration, The loss function is The rate of change in training iterations, , and Respectively Second and The training difference of training iterations, For the The gradient change rate in the training iteration, , and Respectively Second and The gradient value of the training iteration, is a constant.

4. The method for evaluating the quality of labeled data based on active learning according to claim 1, characterized in that: Obtain unlabeled task data and assign the task data to annotators based on their expertise. The annotators annotate the data to obtain a real-time labeled data set, including: According to the data that the annotator is good at, the matching score of the annotator is obtained. The calculation formula of the matching score is: ， in, For the annotator For The matching scores of the data types are: For the annotator In the The average historical annotation accuracy of the data types, For the annotator In the The number of historical annotations of each type of data, For the annotator For The labeling efficiency of different types of data, is the normalization coefficient, is the coefficient; According to the matching scores of the annotators, the task data is assigned to the annotators in the following way: ， in, For the Whether the task data of this type is assigned to the annotator As a result, For the annotator For The matching scores of the data types are: For the A marker, is the total number of annotators; When the annotator is assigned task data, he annotates the task data to obtain a real-time annotated data set.

5. The method for evaluating the quality of labeled data based on active learning according to claim 1, characterized in that: Average historical annotation accuracy The calculation formula is: ， in, For the annotator In the The average historical annotation accuracy of the data types, For the annotator In the The number of historical annotations for each type of data, For the annotator In the Type of data The historical annotation accuracy of historically annotated data; Labeling efficiency The calculation formula is: ， in, For the annotator For The labeling efficiency of different types of data, For the annotator In the The average historical annotation accuracy of the data types, For the annotator Complete The average time of the data type, and is the coefficient.

6. The method for evaluating the quality of labeled data based on active learning according to claim 1, characterized in that: The calculation formula for the consistency of annotation data is: ， in, For the Type task data The consistency of the data annotation results, For the The number of type annotation task data, For the The number of annotators who annotate the task data of type, For the The weight of the annotator, is the similarity function, , For the The annotator The type annotation task The labeling results given by the data For the The annotator The type annotation task The labeling results given by the data, and , As parameters, To adjust the function, , For the Parameters of the labeled data, as a parameter.

Images