Intelligent data labeling method based on active learning and uncertainty quantification

Abstract

The invention relates to the technical field of deep learning, and further relates to an intelligent data labeling method based on active learning and uncertainty quantization, and the method comprises the following steps: 1, obtaining a fusion uncertainty score of each unlabeled sample in a to-be-labeled data set; 2, constructing a standardized feature matrix based on the feature embedding vectors of all the unlabeled samples, and selecting a preset number of unlabeled samples in a descending order according to enhanced scores to form a to-be-labeled sample set of the current batch; and step 3, sequentially presenting the to-be-labeled sample set of the current batch to a manual labeling person according to the enhanced scores from low to high, collecting labeling results, and repeatedly executing the step 1 to the step 3 until a termination condition is met. According to the method, low-cost automatic interception of annotation noise is realized, and continuous and reliable improvement of model performance in an active learning iteration process is guaranteed through a stable increment updating mechanism.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology, specifically relating to an intelligent data annotation method based on active learning and uncertainty quantification. Background Technology

[0002] With the widespread application of deep learning technology in fields such as computer vision, natural language processing, and medical image analysis, high-quality labeled data has become one of the core bottlenecks restricting model performance. In practical engineering scenarios, the scale of the original dataset to be labeled often reaches hundreds of thousands or even millions of samples, while manual labeling has high unit cost and long cycle. How to maximize model performance within a limited labeling budget has become a core issue of common concern to academia and industry.

[0003] Traditional data annotation methods employ random or uniform sampling strategies, randomly selecting a number of samples from the unlabeled dataset for manual annotation, and then adding the annotated results to the training set for model training. The fundamental flaw of this strategy lies in ignoring the varying contributions of different samples to model learning: a large number of samples located in the core regions of the known distribution have extremely limited impact on model improvement, while a few high-value samples located at class boundaries or in remote areas of the feature space are often overlooked. Research shows that, with the same number of annotations, strategically selecting high-information samples for annotation can increase the model's performance on downstream tasks by 30% to 50% compared to random sampling. This difference is particularly significant in scenarios with extremely limited annotation budgets.

[0004] To address these issues, active learning methods have been introduced into the data labeling process. The core idea of ​​active learning is to allow the model to actively participate in the sampling process, prioritizing the submission of samples with the greatest information gain for the model's current state. Early active learning methods were primarily based on uncertainty sampling, including strategies such as minimum confidence sampling, marginal sampling, and information entropy sampling. These methods use the distribution of the model's predicted probabilities as a measure of uncertainty, prioritizing the labeling of samples with the flattest predicted probability distribution (i.e., the model's greatest uncertainty). However, traditional uncertainty sampling methods have a serious inherent flaw: they rely on the model's point estimate of the predicted probability of the sample output, which is essentially a deterministic measure and cannot distinguish between "cognitive uncertainty caused by model parameter uncertainty" and "random uncertainty caused by the ambiguity of the sample's class boundaries." For out-of-distribution samples far from the training distribution, deterministic neural networks often give highly confident but incorrect predictions (i.e., predicted probabilities concentrated in a certain class), making it impossible for traditional uncertainty sampling methods to identify these most deserving samples, resulting in systematic bias in sampling.

[0005] To address the difficulty in separating cognitive uncertainty from stochastic uncertainty, uncertainty quantification methods based on Bayesian deep learning have gained increasing attention. The BALD method measures cognitive uncertainty by calculating the mutual information between model predictions and model parameters. Theoretically, it can effectively separate the contributions of both types of uncertainty, thus more accurately identifying the samples most valuable for model parameter learning. However, the BALD method also faces challenges in engineering implementation: to approximate Bayesian inference, Monte Carlo stochastic inference methods are typically used, requiring multiple repeated forward propagations during the inference phase, resulting in significant computational overhead. More importantly, the BALD method still relies on the model's predicted output for samples; for samples deviating from the known distribution in the feature space, the reliability of the model prediction itself is questionable, leading to a decrease in the estimation accuracy of the BALD score. Meanwhile, the DUQ method, for detecting out-of-distribution samples, determines whether a sample deviates from the known distribution by measuring the distance between the sample's feature embedding vector and the centroid vectors of each class, effectively capturing the out-of-distribution uncertainty missed by the BALD method. However, when used independently, the DUQ method cannot reflect uncertainty at the model parameter level, and its sampling ability is limited for samples within the known distribution but with highly uncertain model parameters. In the existing technology, the BALD method and the DUQ method are usually used as independent sampling strategies. There is a lack of a systematic solution to effectively integrate the two under a unified dimensional framework, which makes it difficult for the sampling strategy to take into account both cognitive uncertainty and out-of-distribution uncertainty at the same time. Summary of the Invention

[0006] The main objective of this invention is to provide an intelligent data annotation method based on active learning and uncertainty quantification. Under the same annotation budget, it significantly improves the information contribution efficiency of labeled samples to model learning, effectively reduces the redundancy of batch sampling, realizes low-cost automatic interception of annotation noise, and ensures the continuous and reliable improvement of model performance during active learning iteration through a stable incremental update mechanism.

[0007] To solve the above problems, the technical solution of the present invention is implemented as follows:

[0008] The intelligent data annotation method based on active learning and uncertainty quantification includes the following steps:

[0009] Step 1: For each unlabeled sample in the dataset to be labeled, calculate the BALD uncertainty score and DUQ uncertainty score using a Bayesian deep neural network. Perform intra-batch normalization on the BALD uncertainty score and DUQ uncertainty score respectively, and take the larger value to obtain the fusion uncertainty score of each unlabeled sample.

[0010] Step 2: Construct a standardized feature matrix based on the feature embedding vectors of all unlabeled samples, calculate the sample covariance matrix corresponding to the standardized feature matrix and perform eigenvalue decomposition, identify anomalous feature values ​​based on the upper bound of the Marchenko-Pastur distribution in random matrix theory, calculate the spectral anomaly response value of each unlabeled sample, and mark unlabeled samples whose spectral anomaly response value exceeds the mean of the spectral anomaly response values ​​of all unlabeled samples as spectrally significant samples. The enhancement score of spectrally significant samples is the sum of the fusion uncertainty score and the spectral anomaly response value. The enhancement score of non-spectrally significant samples is the fusion uncertainty score. Select a preset number of unlabeled samples in descending order of enhancement score to form the current batch of unlabeled sample set.

[0011] Step 3: Present the current batch of samples to be labeled to human annotators in order of enhancement score from low to high and collect the labeling results. Calculate the confidence score for each labeling result using the meta-learning labeling quality assessment sub-network. For labeling results with confidence scores lower than a preset threshold, trigger a secondary manual review mechanism. Add all confirmed labeled samples to the training set and perform incremental updates on the Bayesian deep neural network. Repeat steps 1 to 3 until the termination condition is met.

[0012] Furthermore, the calculation process of the BALD uncertainty score in step 1 is as follows: During the inference phase, the Dropout layer of the Bayesian deep neural network is kept active, and 50 independent forward propagations are performed on each unlabeled sample to obtain 50 sets of predicted probability outputs corresponding to each unlabeled sample; the arithmetic mean of the 50 sets of predicted probability outputs is taken to obtain the average prediction distribution of each unlabeled sample; the information entropy of the average prediction distribution of each unlabeled sample and the arithmetic mean of the information entropy of the 50 sets of predicted probability outputs of each unlabeled sample are calculated to obtain the mean conditional entropy of each unlabeled sample; the information entropy of the average prediction distribution of each unlabeled sample is subtracted from the mean conditional entropy to obtain the original BALD score of each unlabeled sample.

[0013] Further, the calculation process of the DUQ uncertainty score in step 1 is as follows: Each unlabeled sample is passed through the feature extraction layer of a Bayesian deep neural network to output the feature embedding vector of each unlabeled sample; for each category in the training set, the arithmetic mean of the feature embedding vectors of all labeled samples belonging to each category is taken to obtain the centroid vector of each category; the Euclidean distance between the feature embedding vector of each unlabeled sample and the centroid vector of each category in the training set is calculated, and the minimum value of the corresponding Euclidean distances of all categories is taken as the minimum Euclidean distance of each unlabeled sample; the minimum Euclidean distance of each unlabeled sample is divided by the radial basis function bandwidth parameter, and the exponential function value with the natural constant as the base is taken; then the obtained exponential function value is subtracted from 1 to obtain the original DUQ score of each unlabeled sample.

[0014] Furthermore, the batch normalization process in step 1 is as follows: For the raw BALD scores of all unlabeled samples, subtract the minimum value among all raw BALD scores of all unlabeled samples from the raw BALD score of each unlabeled sample, and then divide by the difference between the maximum and minimum values ​​of the raw BALD scores of all unlabeled samples to obtain the BALD normalized score of each unlabeled sample; For the raw DUQ scores of all unlabeled samples, subtract the minimum value among all raw DUQ scores of all unlabeled samples from the raw DUQ score of each unlabeled sample, and then divide by the difference between the maximum and minimum values ​​of the raw DUQ scores of all unlabeled samples to obtain the DUQ normalized score of each unlabeled sample; Take the larger value between the BALD normalized score and the DUQ normalized score of each unlabeled sample as the fusion uncertainty score of each unlabeled sample.

[0015] Furthermore, the construction process of the standardized feature matrix and sample covariance matrix in step 2 is as follows: Arrange the feature embedding vectors of all unlabeled samples by rows to construct a feature matrix. The number of rows in the feature matrix is ​​the total number of unlabeled samples, and the number of columns is the dimension of the feature embedding vectors. Calculate the column mean of the feature matrix column by column. Subtract the corresponding column mean from each element in each column of the feature matrix, and then divide each element in each column by the corresponding column standard deviation to obtain the standardized feature matrix. Multiply the transpose of the standardized feature matrix by the standardized feature matrix and divide by the total number of unlabeled samples to obtain the sample covariance matrix. Perform eigenvalue decomposition on the sample covariance matrix to obtain the eigenvalue sequence composed of all eigenvalues ​​and the eigenvector matrix corresponding to each eigenvalue.

[0016] Furthermore, in step 2, the process of determining the upper bound of the Marchenko-Pastur distribution and identifying anomalous eigenvalues ​​is as follows: calculate the ratio of the number of rows to the number of columns of the standardized feature matrix, sum the arithmetic square root of the ratio to 1, and then square the result to obtain the theoretical noise upper bound; mark the eigenvalues ​​in the eigenvalue sequence that exceed the theoretical noise upper bound as anomalous eigenvalues.

[0017] Furthermore, the calculation of the spectral anomaly response value and the determination of the current batch of unlabeled sample set in step 2 are as follows: For each anomaly feature value, the feature vector corresponding to each anomaly feature value is extracted from the feature vector matrix, and the absolute value of the projection of the standardized feature embedding vector of each unlabeled sample onto the feature vector direction corresponding to each anomaly feature value is calculated; the arithmetic mean of the absolute values ​​of the projections of each unlabeled sample onto the feature vector directions corresponding to all anomaly feature values ​​is taken to obtain the spectral anomaly response value of each unlabeled sample; unlabeled samples whose spectral anomaly response value exceeds the mean of the spectral anomaly response values ​​of all unlabeled samples are marked as spectrally significant samples; the enhancement score of spectrally significant samples is the sum of the fusion uncertainty score of spectrally significant samples and the spectral anomaly response value; the enhancement score of non-spectrally significant samples is the fusion uncertainty score of non-spectrally significant samples; all unlabeled samples are sorted in descending order according to the enhancement score, and the top 300 unlabeled samples are taken to form the current batch of unlabeled sample set.

[0018] Furthermore, in step 3, the process of calculating the confidence score through the meta-learning annotation quality assessment sub-network is as follows: the standardized feature embedding vector of each submitted annotation result sample is concatenated with the fusion uncertainty score to obtain the concatenated vector of each submitted annotation result sample; the concatenated vector of each submitted annotation result sample is used as the input of the meta-learning annotation quality assessment sub-network, which outputs a 2-dimensional vector. Softmax operation is performed on the 2-dimensional vector, and the value of the first dimension in the 2-dimensional output is the confidence score of the corresponding sample annotation result; annotation samples with a confidence score lower than 0.6 are added to the review queue and re-annotated independently by a second human annotator, and the result with the higher confidence score between the two annotation results is taken as the final annotation result; annotation samples with a confidence score not lower than 0.6 directly use the initial annotation result.

[0019] Furthermore, the training process of the meta-learning annotation quality evaluation sub-network is as follows: Labeled samples are extracted from the training set, and directional class flipping is performed according to the class pairs with the highest confusion predicted by the Bayesian deep neural network on similar feature samples to construct mislabeled samples; the quality label of the correctly labeled samples is set to 1, and the quality label of the mislabeled samples is set to 0; the concatenation vector of the standardized feature embedding vector and the fusion uncertainty score of each correctly labeled sample and the concatenation vector of the standardized feature embedding vector and the fusion uncertainty score of each mislabeled sample are used as inputs, and the quality label is used as the supervision signal. Gradient descent training is performed using cross-entropy loss until the meta-learning annotation quality evaluation sub-network converges.

[0020] Furthermore, the incremental update process and termination condition in step 3 are as follows: freeze the feature extraction layer parameters of the Bayesian deep neural network, and only perform mini-batch stochastic gradient descent update on the classification output layer of the Bayesian deep neural network with a mini-batch consisting of all confirmed labeled samples in the current batch; simultaneously perform incremental update on the meta-learning annotation quality evaluation sub-network with all confirmed labeled samples in the current batch; after each incremental update, for each category in the training set, recalculate the arithmetic mean of the feature embedding vectors of all labeled samples belonging to each category to update the centroid vector of each category; the termination condition is that the annotation budget is exhausted, or the relative change rate of the mean of the fusion uncertainty score of all unlabeled samples in three consecutive rounds is less than 5%.

[0021] The intelligent data annotation method based on active learning and uncertainty quantification described in this invention has the following beneficial effects: This invention overcomes the limitation of existing active learning methods that rely on a single uncertainty metric. The BALD method identifies samples with the highest cognitive value from the perspective of model parameter uncertainty, while the DUQ method identifies out-of-distribution samples that deviate from the known distribution from the perspective of feature space distribution distance. These two paths capture information that existing technologies can only cover one aspect of. After eliminating the dimensional differences between the two scores through intra-batch normalization, fusion is performed, enabling the fused uncertainty score to effectively perceive high-value samples of all types in the feature space. This avoids the problem of systematically missing certain important samples under a single metric strategy, fundamentally improving the comprehensiveness and reliability of the sampling strategy.

[0022] This invention introduces the upper bound of the Marchenko-Pastur distribution from random matrix theory into the active learning sampling process. By performing spectral analysis on the sample covariance matrix composed of the feature embedding vectors of all unlabeled samples, it identifies statistically significant low-rank structural directions in the data. Based on this, it enhances the fusion uncertainty score of spectrally significant samples with strong responses in these structural directions. This mechanism ensures that the final batch of samples to be labeled maintains a high level of uncertainty while taking into account the global structural coverage of the feature space. It effectively alleviates the "pattern collapse" problem commonly found in traditional uncertainty sampling strategies, making the coverage of new information in the feature space more balanced in each round of labeling, thereby accelerating the improvement of model performance with each round of labeling.

[0023] The meta-learning annotation quality assessment sub-network of this invention constructs training data using a directional category flipping method based on model confusion pairs. This ensures that the learned error annotation patterns highly match the category pairs most easily confused by annotators in real-world scenarios, significantly outperforming the method of constructing noisy annotations using random flipping. Combined with a progressive annotation order design, samples are presented sequentially from low enhancement scores to high enhancement scores, guiding annotators to process difficult samples only after establishing a sufficient category cognitive framework. This further reduces the incidence of annotation errors from a human-machine collaborative perspective. Annotation results with confidence scores below a threshold trigger independent secondary manual review, effectively intercepting noisy labels with low additional cost within a single annotator workflow, thus ensuring the quality of annotated data entering the training set. Attached Figure Description

[0024] Figure 1 A schematic diagram illustrating the core principle of the BALD and DUQ dual-path uncertainty quantification mechanism provided in this embodiment of the invention;

[0025] Figure 2 A schematic diagram comparing the feature space coverage effect of the enhanced score selection mechanism and the pure fusion uncertainty selection mechanism provided in the embodiments of the present invention;

[0026] Figure 3 This is a schematic diagram illustrating the variation of model classification accuracy with the number of labeled samples under five different active learning sampling strategies, and a quantitative comparison of the number of labels required to achieve the same accuracy target, provided by embodiments of the present invention.

[0027] Figure 4 This diagram illustrates the changing patterns of three key performance indicators of the meta-learning annotation quality evaluation subnetwork provided in this embodiment of the invention during 10 consecutive rounds of active learning iterations. Detailed Implementation

[0028] The intelligent data annotation method based on active learning and uncertainty quantification includes the following steps:

[0029] Step 1: For each unlabeled sample in the dataset to be labeled, calculate the BALD uncertainty score and DUQ uncertainty score using a Bayesian deep neural network. Perform intra-batch normalization on the BALD uncertainty score and DUQ uncertainty score respectively, and take the larger value to obtain the fusion uncertainty score of each unlabeled sample.

[0030] Step 2: Construct a standardized feature matrix based on the feature embedding vectors of all unlabeled samples, calculate the sample covariance matrix corresponding to the standardized feature matrix and perform eigenvalue decomposition, identify anomalous feature values ​​based on the upper bound of the Marchenko-Pastur distribution in random matrix theory, calculate the spectral anomaly response value of each unlabeled sample, and mark unlabeled samples whose spectral anomaly response value exceeds the mean of the spectral anomaly response values ​​of all unlabeled samples as spectrally significant samples. The enhancement score of spectrally significant samples is the sum of the fusion uncertainty score and the spectral anomaly response value. The enhancement score of non-spectrally significant samples is the fusion uncertainty score. Select a preset number of unlabeled samples in descending order of enhancement score to form the current batch of unlabeled sample set.

[0031] Step 3: Present the current batch of samples to be labeled to human annotators in order of enhancement score from low to high and collect the labeling results. Calculate the confidence score for each labeling result using the meta-learning labeling quality assessment sub-network. For labeling results with confidence scores lower than a preset threshold, trigger a secondary manual review mechanism. Add all confirmed labeled samples to the training set and perform incremental updates on the Bayesian deep neural network. Repeat steps 1 to 3 until the termination condition is met.

[0032] In real-world engineering scenarios, the core bottleneck of data annotation is not the ease of use of annotation tools, but rather how to achieve the greatest model performance improvement with minimal manual labor costs. In large-scale unlabeled datasets, the vast majority of samples contribute highly redundantly to the model—they fall within distribution regions that the model has already fully learned, and labeling them yields almost no new information gain. Therefore, the basic premise of this approach is to proactively identify the samples with the greatest informational value to the current model from the unlabeled samples before each round of annotation, concentrating the limited annotation budget on the most worthwhile parts, thereby achieving a systematic improvement in annotation efficiency.

[0033] Steps 1 and 2 are designed to achieve this goal: Step 1 quantifies the information value of each unlabeled sample to the model, while Step 2 further introduces structural analysis of the feature space to ensure that the final selected batch of samples to be labeled not only has high information content but also sufficient diversity and structural coverage in the feature space. The two steps work together to form the core of the active learning mechanism of this scheme.

[0034] The goal of Step 1 is to assign a fusion uncertainty score to each unlabeled sample to quantify its potential informational value to the current model. To this end, this scheme designs two parallel uncertainty calculation paths, based on the BALD and DUQ methods respectively. These two methods characterize the uncertainty of samples from different perspectives and are ultimately fused into a unified score. The core idea of ​​the BALD method is that when the model's prediction of a sample highly depends on the specific values ​​of the parameters, it indicates that the model has strong cognitive uncertainty about that sample. Labeling the sample can effectively reduce the model's uncertainty about the parameter space, thereby bringing the maximum information gain. Bayesian neural networks can naturally express the uncertainty distribution of parameters, but in practical engineering, accurate Bayesian inference is computationally very expensive. MCDropout is a widely adopted approximation scheme: during the inference phase, the Dropout layer is kept active (instead of being turned off as in ordinary inference). Each forward propagation is equivalent to randomly sampling a model instance from the posterior distribution of the parameters; repeating this process multiple times approximates the posterior prediction distribution.

[0035] Specifically, for each unlabeled sample, the Dropout layer of the Bayesian deep neural network is kept active during the inference phase, and 50 independent forward propagations are performed on that sample to obtain 50 sets of predicted probability outputs. The reason for choosing 50 times is that experiments show that when the number of samples exceeds 30, the estimated variance of the approximate posterior prediction distribution tends to stabilize, and the marginal benefit of further increasing the number of samples decreases rapidly. 50 times achieves a good balance between estimation accuracy and computational cost. For scenarios with extremely limited computational resources, the number of samples can be appropriately reduced to 20 times; for scenarios requiring extremely high accuracy in screening small-scale key samples, it can be appropriately increased to 100 times.

[0036] The arithmetic mean of the 50 predicted probability outputs is taken to obtain the average predicted distribution of the sample, denoted as . ,in This indicates that there are currently no labeled samples. This represents the category label. The average prediction distribution reflects the model's overall judgment of which category the sample belongs to after integrating the model parameters.

[0037] The information entropy of the average prediction distribution is denoted as ,Right now ;in The total number of categories, The th in the average prediction distribution The predicted probability for each category. This measure represents the overall uncertainty of the model regarding the sample after integrating the parameters, including both cognitive and stochastic uncertainties.

[0038] Meanwhile, the predicted probability outputs for each of the 50 forward propagations are... ( Calculate the information entropy respectively Then, take the arithmetic mean of the 50 results to obtain the mean conditional entropy: The conditional entropy mean measures the average uncertainty of the model for a particular sample under each specific parameter instance. It reflects the random uncertainty of the data itself, that is, the part of uncertainty that cannot be eliminated even if the model parameters are known (such as the ambiguity of the sample itself being at the class boundary).

[0039] The raw BALD score is defined as the difference between the mean information entropy and the mean conditional entropy of the average prediction distribution. This difference is actually the mutual information between the model prediction and the parameters, directly measuring the magnitude of cognitive uncertainty. Samples with high BALD scores indicate significant discrepancies in the predictions made by different parameter instances—these are precisely the samples most worthy of annotation: annotating them helps the model make more definitive judgments in the parameter space, effectively reducing cognitive uncertainty. Conversely, if a sample's BALD score is close to 0, it means that regardless of changes in the model parameters, the predictions tend to be consistent, and the information gain from annotating this sample is extremely limited.

[0040] However, the BALD method has an inherent limitation: it relies on the model's predicted output for samples. For samples that are far from the training data distribution (i.e., out-of-distribution samples), the model often makes seemingly confident (i.e., low-entropy) incorrect predictions, which makes BALD unable to identify the high uncertainty of such out-of-distribution samples. The DUQ method was introduced to overcome this deficiency.

[0041] The DUQ method does not rely on the model's predicted probabilities. Instead, it determines whether a sample falls within the coverage of a known data distribution by measuring the distance between the sample's feature embedding vector and the center of the feature space for each class. The intuition is that a well-trained feature extraction layer will cluster the feature embedding vectors of similar samples in specific regions of the feature space. If the feature embedding vector of an unlabeled sample is far from the clustered regions of all classes, it indicates that the sample is in a "no-man's land" in the feature space; the model has never seen similar data, its uncertainty is extremely high, and it should be labeled first.

[0042] Each unlabeled sample is passed through the feature extraction layer of a Bayesian deep neural network, outputting a feature embedding vector for each unlabeled sample, denoted as . ,in Dimension is the feature embedding vector. This applies to each category in the training set. ( ), will belong to the category The category is obtained by taking the arithmetic mean of the feature embedding vectors of all labeled samples. The centroid vector: ;in For categories in the training set The number of labeled samples, For category The Middle The feature embedding vector of each labeled sample. The centroid vector represents the distribution center of the class in the feature space and is a reference for determining which known class an unlabeled sample is closest to.

[0043] Calculate the feature embedding vector for each unlabeled sample With the centroid vector of each category Euclidean distance between The minimum Euclidean distance for a sample is taken as the minimum Euclidean distance among all categories, denoted as . The larger the minimum Euclidean distance, the farther the sample is from the feature space center of all known categories, meaning it is more likely to be an out-of-distribution sample and has higher uncertainty.

[0044] The minimum Euclidean distance is transformed into the raw DUQ score through radial basis function mapping: ;in The radial basis function bandwidth parameter controls the sensitivity of the distance-to-score mapping. When Much larger When the original DUQ score approaches 1, it indicates that the sample deviates significantly from the known distribution; when much smaller When the original DUQ score approaches 0, it indicates that the sample is in the core region of the known distribution, and the model has sufficient coverage of it. Bandwidth parameter Typically, this can be set to the median of the Euclidean distances from all labeled samples in the training set to their respective class centroids. This adaptive setting allows the distribution of DUQ scores to maintain a certain stability with respect to the current training set size and feature space scale. In terms of feature space dimensions... In higher-dimensional scenarios (e.g., exceeding 256 dimensions), the distribution of Euclidean distance tends to be concentrated due to the curse of dimensionality. In this case, it is advisable to perform principal component analysis to reduce the dimensionality of the feature embedding vectors before calculating the Euclidean distance, retaining the principal components whose cumulative variance contribution rate reaches 95%, and then calculating the Euclidean distance between the dimensionality-reduced feature embedding vectors and the dimensionality-reduced centroid vectors to alleviate the problem of distance metric degradation in high-dimensional spaces.

[0045] The dimensions and numerical ranges of the BALD raw score and the DUQ raw score differ significantly: the BALD raw score depends on the number of categories. And the sharpness of the predicted distribution, in When the value is large, it may reach a high value; the original DUQ score, after radial basis function transformation, is naturally between 0 and 1, but its actual distribution is affected by the bandwidth parameter. The impact is significant. If the larger value is directly chosen for both, it is highly likely that one score will dominate in terms of magnitude for a long period of time, causing the information from the other to become completely invalid. For example, when there are many categories, the original BALD score may be much higher than the original DUQ score. In this case, the fusion operation is actually reduced to sorting only using the BALD score, and the ability to detect out-of-distribution problems is completely lost.

[0046] To eliminate dimensional differences, intra-batch normalization needs to be performed on both the raw BALD scores and the raw DUQ scores. For all unlabeled samples, the maximum raw BALD score is recorded as [value missing]. The minimum value is Then the BALD normalized score for each unlabeled sample is: For all unlabeled samples, the raw DUQ scores were used to obtain the normalized DUQ score for each unlabeled sample in the same manner. After normalization, the scores for both paths are between 0 and 1, which meets the prerequisite for direct comparison.

[0047] refer to Figure 1 The left subplot presents the distribution of all unlabeled samples in a two-dimensional scatter plot, defined by the BALD and DUQ normalized scores. The horizontal axis represents the BALD normalized score for each sample, and the vertical axis represents the DUQ normalized score. Both scores have undergone intra-batch max-min normalization and range from 0 to 1. A diagonal dashed line with a slope of 1 and an intercept of 0 marks the boundary where the BALD and DUQ normalized scores are equal, i.e., satisfying the condition... The sample trajectory. The region above the diagonal corresponds to samples with DUQ normalized scores higher than BALD normalized scores. The feature embedding vectors of these samples deviate significantly from the known class centroid vectors, belonging to the type dominated by out-of-distribution uncertainty. The region below the diagonal corresponds to samples with BALD normalized scores higher than DUQ normalized scores. The cognitive uncertainty of the model parameters dominates for these samples. In two-dimensional space, the samples are naturally divided into four typical regions: high BALD low DUQ region (dominated by cognitive uncertainty), low BALD high DUQ region (dominated by out-of-distribution uncertainty), high BALD high DUQ region (double high-value samples), and low BALD low DUQ region (low-information-value samples). The fusion uncertainty score is defined as... The larger of the two scores is taken, meaning that if a sample exhibits high uncertainty in any dimension, its uncertainty score will be higher, thus giving it priority for inclusion in the annotation batch. This maximum-value fusion strategy effectively captures high-value samples located on either side of the diagonal, avoiding the systematic omissions of single-measure strategies for certain sample types. A median line on each side further divides the scatter plot into four quadrants. The figure clearly shows that the four sample groups cluster within their respective quadrants, with high-BALD and high-DUQ dual-high-value samples concentrated in the upper right quadrant. These samples possess both high cognitive uncertainty and deviate from the known data distribution, making them the highest priority for annotation.

[0048] The right-hand subplot quantitatively shows the number of MCDropout samples. The impact on the variance of BALD score estimation. The horizontal axis represents the number of samples. The value range covers 5 to 100 times; the vertical axis represents the variance of the BALD score estimate, reflecting the accuracy of the approximation of the Bayesian posterior prediction distribution under a finite number of samplings. The curve shows that... The trend of increasing and monotonically decreasing, and in The decrease was significant when the value was small, and then... After increasing, it gradually tends to stabilize. Specifically, when When the value increases from 5 to 20, the estimated variance decreases significantly; when After more than 30 iterations, the reduction in variance from further increasing the number of samples becomes relatively limited. The figure also shows the quantile intervals between the 25th and 75th percentiles in multiple repeated experiments. When the value is small, the width is large, reflecting the estimation instability caused by insufficient sampling; as... As the range increases, the quantile interval gradually narrows, indicating that the estimation results tend to be robust. This is indicated by the vertical dashed line in the figure. The estimated variance corresponding to this recommended number of sampling times has entered the convergence plateau region. At this point, the marginal benefit of further increasing the number of sampling times is extremely limited, while the computational cost of 50 samplings is within an acceptable range in engineering practice. This analysis provides a quantitative basis for selecting 50 independent forward propagations as the sampling number for BALD score estimation in this scheme.

[0049] The fusion strategy uses the larger value for each sample: ;in Unlabeled samples The uncertainty scores are fused. A larger value is chosen instead of summation or a weighted average because the physical meanings of the two uncertainty measures are fundamentally different: high-scoring BALD samples are those with uncertain model parameters, while high-scoring DUQ samples are those whose features deviate from a known distribution; the two are not additive. For a given sample, if it is judged to have high uncertainty in any dimension, it should be considered to have high annotation value; taking a larger value can capture this "extreme in any dimension" and prevent important samples from being missed due to the dilution of the two scores.

[0050] Optionally, in certain specific scenarios, such as when the training set size is extremely small (less than 5 samples per class), the reliability of the class centroid vector estimation is poor. In this case, the weight of the DUQ normalization score can be temporarily reduced. Specifically, the fusion uncertainty score is changed to a weighted average of the BALD normalization score and the DUQ normalization score, with the weight of the BALD normalization score set to 0.7 and the weight of the DUQ normalization score set to 0.3. After the training set size grows to at least 10 samples per class, the fusion strategy of taking the larger value is restored.

[0051] After completing step 1, each unlabeled sample receives a fusion uncertainty score. Directly sorting the samples by this score and selecting the top few is the most straightforward active learning strategy. However, this strategy suffers from a classic "pattern collapse" problem: samples with high fusion uncertainty scores tend to cluster in specific difficult regions in the feature space, such as near the boundary between two easily confused categories. If sorted solely by this score, the selected 300 samples may mostly come from the same difficult region and be highly similar to each other. While the model's performance in that region may improve after labeling, the improvement in uncertainty in other uncovered regions is extremely limited, significantly reducing overall labeling efficiency. Step 2 is designed to address this problem.

[0052] The core idea of ​​Step 2 is: based on the existing fusion uncertainty scores, by analyzing the spectral structure of the matrix formed by the feature embedding vectors of all unlabeled samples, identify those "structurally significant" samples that make important contributions to the global structure of the feature space. Enhance the fusion uncertainty scores of these samples, thereby naturally guiding the sampling selection towards a more uniform feature space coverage direction in the final batch selection. The idea of ​​spectral monitoring originates from random matrix theory: for a matrix composed of purely random vectors, the eigenvalue distribution of its covariance matrix follows a Marchenko-Pastur distribution; if abnormally large eigenvalues ​​significantly deviate from this purely random baseline appear in the actual eigenvalue distribution, it indicates the existence of low-rank but strongly structured signal directions in the data. Identifying samples with strong responses in these directions is precisely finding the most representative and diverse samples in the feature space.

[0053] Specifically, the feature embedding vectors of all unlabeled samples are arranged row by row to construct a feature matrix. ,in This represents the total number of unlabeled samples. is the dimension of the feature embedding vector.

[0054] Before performing random matrix analysis on the feature matrix, column centering and scale normalization must be performed. These two preprocessing steps are necessary because the Marchenko-Pastur theorem requires that matrix elements have zero mean and a consistent variance scale. If there are significant mean shifts or variance differences between different columns of the feature matrix, the eigenvalue distribution of the covariance matrix will deviate from the Marchenko-Pastur benchmark, rendering the theoretical noise upper bound meaningless and thus rendering the identification of outlier eigenvalues ​​ineffective.

[0055] For the characteristic matrix The List( ), calculate column mean Subtract the corresponding column mean from each element in each column; then calculate the centered standard deviation of each column. Divide each column by its corresponding standard deviation to obtain the standardized feature matrix. The columns must have a mean of 0 and a variance of 1. If the standard deviation of a column is less than 1... If the value is a constant, then all its elements are set to 0 to avoid division by zero errors.

[0056] Based on the standardized feature matrix Calculate the sample covariance matrix: ;in , This is the transpose of the standardized feature matrix. For the sample covariance matrix... Perform eigenvalue decomposition to obtain an eigenvalue sequence consisting of all eigenvalues. and the corresponding eigenvector matrix ,in Eigenvalues The corresponding unit eigenvector.

[0057] According to the Marchenko-Pastur distribution, when the matrix elements satisfy the condition of independent and identically distributed zero mean and unit variance, the eigenvalues ​​of the sample covariance matrix are distributed in... and Both tend to infinity and the ratio In the limiting condition approaching the stability constant, it converges to the Marchenko-Pastur distribution. The supporting upper bound of the Marchenko-Pastur distribution is: ;in This is the ratio of the number of columns to the number of rows in the standardized feature matrix. Theoretical upper bound for noise. This means that if the data contains no low-rank structured signals and only random noise, then the largest eigenvalue of the sample covariance matrix should not exceed [a certain value]. Anything exceeding The eigenvalues ​​can all be considered to correspond to statistically significant structural orientations that actually exist in the data, rather than fluctuations of random noise. The values ​​in the eigenvalue sequence exceeding the theoretical noise upper bound are considered... The eigenvalues ​​are labeled as anomalous eigenvalues, and the set of anomalous eigenvalues ​​is denoted as . Its corresponding feature vector is .

[0058] when When the feature dimension is greater than the number of samples, the sample covariance matrix is ​​singular with some eigenvalues ​​being zero. In this case, the applicability of the Marchenko-Pastur theory changes. It is recommended to perform principal component analysis on the standardized feature matrix first, retaining principal components with a cumulative variance contribution rate of 99%, so that the processed matrix satisfies... ( (The dimension is reduced to the desired dimension), and then the sample covariance matrix is ​​reconstructed based on the dimension-reduced feature embedding vectors before performing subsequent analysis. This is done to ensure that the preconditions for the applicability of random matrix theory are met, and to avoid eigenvalue estimation distortion due to matrix singularity.

[0059] After determining the set of anomalous feature values, for each unlabeled sample Calculate its standardized feature embedding vector (i.e., standardized feature matrix) The Middle The absolute value of the projection of the row vector corresponding to each anomalous feature value along the direction of the feature vector is taken as the arithmetic mean of the absolute values ​​of the projections along the directions corresponding to all anomalous feature values, and the spectral anomaly response value of the sample is obtained. ;in The number of abnormal feature values. Standardize the feature embedding vector With feature vectors The inner product (i.e., projection value) of the spectral anomaly response value is the average response intensity of the sample across all statistically significant structural directions in the feature space. Samples with high spectral anomaly response values ​​exhibit strong activation in key structural directions of the feature space, indicating a significant contribution to the structure of the current batch's feature space. Including these samples in the labeled batch helps cover these key directions and improves the feature diversity of the batch. Samples with low spectral anomaly response values ​​contribute less in these directions and are relatively less representative.

[0060] The spectral anomaly response value exceeds the mean of all unlabeled spectral anomaly response values. Unlabeled samples were marked as spectrally significant samples, and the remaining unlabeled samples were marked as spectrally insignificant samples. The rationale for using the mean as the dividing threshold is that the mean divides the spectral anomaly response values ​​of all samples into two groups: above average and below average. Samples above the average level are more representative than typical in the feature space structure and should receive additional incentives in the scoring process.

[0061] The enhancement score for spectral salient samples is defined as the sum of the fusion uncertainty score and the spectral anomaly response value: The enhancement score for non-spectral salient samples directly adopts the fusion uncertainty score: The reason for directly adding the spectral anomaly response value to the fusion uncertainty score, instead of multiplying it by a coefficient, is to ensure that the enhancement magnitude is linearly proportional to the absolute magnitude of the spectral anomaly response value. The stronger the response, the greater the excitation, thus avoiding the introduction of additional hyperparameters.

[0062] It should be noted that the fusion uncertainty score, after intra-batch normalization, falls between 0 and 1, while the scale of the spectral anomaly response value depends on the dimension of the feature embedding vector and the data distribution, resulting in a scale difference between the two. In practice, it is also recommended to perform intra-batch normalization on the spectral anomaly response value across all unlabeled samples (i.e., subtract the minimum value and divide by the difference between the maximum and minimum values, mapping to between 0 and 1), before superposition, to ensure a reasonable proportion of the two contributions and controllable enhancement effects. After normalization, the enhancement scores of spectrally significant samples are between 1 and 2, while those of non-spectrally significant samples are between 0 and 1. Spectrally significant samples naturally have priority in the ranking, while their internal ranking is still determined by the fusion uncertainty score, ensuring that the final selected batches simultaneously consider uncertainty and structural diversity.

[0063] All unlabeled samples are sorted in descending order according to their enhancement scores, and the top 300 unlabeled samples are selected to form the current batch of samples to be labeled. The preset number of 300 is an empirical value determined after comprehensively considering the labor cost of a single round of labeling, the effect of incremental model updates, and the iteration speed of active learning. For scenarios with a small labeling team (e.g., single-person labeling), the preset number can be adjusted to 50 to 100. For industrial scenarios with a large-scale labeling team, the preset number can be appropriately increased to 500 to 1000 to make full use of labeling capacity.

[0064] refer to Figure 2The figure presents the feature space distribution of all unlabeled samples using a two-dimensional principal component projection. The horizontal and vertical axes correspond to the first and second principal components of the feature embedding vectors after dimensionality reduction via principal component analysis, respectively, forming a two-dimensional projection plane of the feature space. All unlabeled samples exhibit a distribution structure centered on eight clusters on this plane, with certain intervals between clusters, representing the distribution positions of different semantic categories or different data subgroups in the feature space. The figure uses two different point shapes to indicate the distribution based solely on the fusion uncertainty score. The samples selected for annotation after being sorted in descending order, and the samples based on the enhancement scores. The selected samples for labeling are arranged in a specific order. As clearly observed in the figure, samples selected based on pure fusion uncertainty are highly concentrated near a few specific clusters, especially in the boundary region of two or three adjacent clusters in the upper left corner of the figure. Almost no selected samples are distributed in other clusters. This phenomenon is a typical "pattern collapse" problem in active learning: samples with high uncertainty are not evenly distributed in the feature space, but are concentrated in a few difficult regions, resulting in high redundancy within the selection batch and insufficient coverage of the rest of the feature space. In contrast, samples selected based on augmentation scores are distributed across all eight clusters, with several samples selected into the current batch within each cluster. The distribution of samples in the feature space is more uniform, and the coverage is significantly expanded. Augmentation scores By combining the fusion uncertainty score with the spectral anomaly response value The superposition mechanism provides additional incentives to spectrally significant samples that exhibit strong projection responses along the eigenvector directions corresponding to anomalous eigenvalues ​​in the sample covariance matrix. This ensures that samples with strong structural representativeness, but not necessarily the highest uncertainty, can also obtain sufficiently high priority in the ranking and be included in the current annotation batch. Coverage comparisons show that the spectral monitoring mechanism essentially superimposes diversity constraints on uncertainty-driven sampling, enabling each annotation batch to simultaneously consider both information content and feature space distribution balance, thereby improving overall annotation efficiency.

[0065] refer to Figure 3The left subplot presents a comparison of five strategies in the form of learning curves. The horizontal axis represents the cumulative number of labeled samples, increasing from 50 to 1500; the vertical axis represents the corresponding classification accuracy. The five curves correspond to random sampling, entropy sampling, BALD sampling only, DUQ sampling only, and the method of this invention, respectively. All five curves exhibit a saturation curve pattern where accuracy continuously increases with the number of labeled samples, and the rate of increase gradually narrows, consistent with the general laws of active learning. When the number of labeled samples is small (less than 200), the accuracy difference between the methods is relatively limited because the model foundation of any sampling strategy is relatively weak under conditions of a very small number of labeled samples. As the number of labeled samples increases, the gap between the methods gradually widens and tends to stabilize. The curve corresponding to the method of this invention remains above the other four curves throughout the entire range of labeled samples, achieving the highest classification accuracy with the same number of labeled samples. In the range of 300 to 700 labeled samples, the accuracy difference between the method of this invention and random sampling is the most significant, demonstrating the advantage of the enhanced score sampling mechanism in the stage of limited labeling budget. BALD sampling and DUQ sampling alone are generally superior to random sampling and entropy sampling, but the difference between them is limited, indicating that single-path uncertainty quantization has a certain gap in sample selection quality compared to two-path fusion quantization. The right-hand subplot presents the number of labeled samples required for each method to achieve the five target accuracy levels of 70%, 75%, 80%, 85%, and 88% in the form of a bar chart. Taking the target accuracy of 80% as an example, random sampling requires approximately 900 to 1000 labeled samples to achieve this, while the method of this invention only requires approximately 500 to 600 labeled samples to achieve the same target, saving approximately 40% to 45% of the number of labeled samples required by random sampling. As the target accuracy increases, the absolute difference in the number of labeled samples required by each method further widens, indicating that the labeling efficiency advantage of the method of this invention is more prominent when pursuing high-precision models. The bar comparisons corresponding to the five sets of target accuracy all show that the method of this invention requires the fewest labeled samples, quantitatively verifying the labeling efficiency improvement effect of the synergistic effect of dual-path uncertainty fusion and spectral monitoring enhancement mechanism.

[0066] The two steps above together accomplish the core task of actively selecting batches of high-value samples from the unlabeled dataset. Step 1, through the fusion of BALD and DUQ dual-path uncertainty quantification and normalization, ensures that the information value of each unlabeled sample is comprehensively evaluated from two complementary dimensions: parameter uncertainty and out-of-distribution deviation. Step 2 analyzes the global structure of the feature space through random matrix spectrum monitoring, incentivizing samples with strong structural representativeness in the scoring process. This further improves the structural coverage and diversity of the unlabeled batch in the feature space while ensuring the uncertainty of sample selection. The synergistic design of these two steps enables this scheme to achieve a more balanced and efficient improvement in model performance compared to traditional active learning methods with the same labeling budget.

[0067] Step 3 is responsible for transforming the high-value unlabeled samples selected in the first two steps into high-quality labeled data. After each round, it drives the model to adaptively update, providing a more accurate uncertainty assessment basis for the next round of active sampling. The design of this step revolves around two core issues: first, how to arrange the sample presentation order to reduce the cognitive burden on annotators and reduce labeling errors; second, how to automatically identify and correct erroneous labels generated during the labeling process to avoid noisy labels contaminating the training set.

[0068] The current batch of samples to be labeled is arranged in ascending order of enhancement score, forming a progressive labeling sequence. Samples are presented to human labelers in this order. This arrangement from low to high enhancement scores is essentially a labeling strategy that progresses from relatively certain samples to relatively ambiguous ones. Samples with low enhancement scores have low fusion uncertainty scores and inconspicuous spectral anomaly responses, indicating that these samples are located in the core region of a known distribution in the feature space. The model's predictions for these samples are relatively consistent, and their category classification is relatively clear. Labelers require less judgment when dealing with these samples, resulting in higher labeling accuracy. Conversely, samples with high enhancement scores are often difficult samples with ambiguous category boundaries and significant deviations in the feature space. Labelers need to make judgments on these difficult samples within the existing category cognitive framework established by the context (i.e., several previously labeled clear samples). This aligns with human cognitive patterns—processing simpler samples first helps labelers form a correct cognitive framework for the current task, leading to significantly better judgment quality when facing difficult samples compared to random presentation. This approach is consistent with the core idea of ​​course learning, the only difference being that the "course" here is designed for human annotators, rather than for models.

[0069] Human annotators complete the category labeling for each presented sample and submit the labeling results. Immediately after submission, the labeling quality is evaluated. Automated quality evaluation relies on a specially designed meta-learning labeling quality evaluation subnetwork. The motivation for introducing this network is that human annotation is not always reliable, especially for difficult samples with high enhancement scores, where the annotator's error rate is often significantly higher than for simple samples. If all labeled results are included in the training set without discrimination, noisy labels will interfere with the incremental updates of the Bayesian deep neural network, causing the model to misjudge the uncertainty of certain regions in the next round of sampling, thus creating a negative cycle of "incorrect labeling leading to a worse model, and a worse model making it even harder to select high-quality samples." The role of the meta-learning labeling quality evaluation subnetwork is to pre-judge the reliability of each labeling result before it enters the training set, intercepting suspicious results and sending them to a review queue, thus controlling the quality of the training data from the source.

[0070] The meta-learning annotation quality evaluation subnetwork is a lightweight binary classification network. Its input is a concatenated vector of the standardized feature embedding vector of each submitted labeled sample and a vector fused with uncertainty scores. Specifically, the standardized feature embedding vector of each submitted labeled sample... (The standardization process is consistent with that used in step 2 when constructing the standardized feature matrix, i.e., the original feature embedding vectors are subjected to the same centering and scaling transformations based on the column mean and column standard deviation calculated in step 2.) and fusion uncertainty score Concatenate the vectors along their dimensions to obtain a vector with dimension [dimensionality]. spliced ​​vector Concatenate vectors The input subnetwork is the meta-learning annotation quality evaluation subnetwork, which outputs a 2D vector. A softmax operation is performed on this 2D vector, and the value of the first dimension in the 2D output is the confidence score of the sample annotation result, denoted as . The closer the confidence score is to 1, the higher the probability that the meta-learning annotation quality evaluation sub-network judges the annotation result as a correct annotation; the closer it is to 0, the more likely it is an incorrect annotation.

[0071] The reason for using both the standardized feature embedding vector and the fused uncertainty score as input, rather than just one, is that they provide complementary information. The standardized feature embedding vector encodes the sample's position in the feature space, reflecting the sample's semantic content; the fused uncertainty score encodes the model's overall uncertainty about the sample, reflecting the sample's difficulty within the current model's knowledge framework. For samples whose feature embedding vectors are far from all class centroid vectors (i.e., high fused uncertainty scores), the annotator's prior probability of error is already higher. The meta-learning annotation quality evaluation sub-network, seeing both the feature embedding vector and the fused uncertainty score simultaneously, can more accurately identify the pattern of "suspicious annotation results on high-difficulty samples," rather than judging solely based on feature space position or uncertainty level.

[0072] The training process of the meta-learning annotation quality evaluation sub-network is key to understanding the creativity of the entire scheme. Annotated samples are extracted from the training set, and these samples undergo targeted class flipping based on the class pair with the highest predicted confusion among samples with similar features predicted by the Bayesian deep neural network, thus constructing mislabeled samples. The "class pair with the highest predicted confusion" refers to the class with the second highest predicted probability after the class with the highest predicted probability in the average prediction distribution of the Bayesian deep neural network. The highest confusion between these two classes means that the model itself has the weakest ability to distinguish between these two types of samples, which is precisely where annotators are most prone to labeling confusion. For a given original class... Given the labeled samples, if the class pair with the highest confusion in the prediction distribution of samples with similar features by the Bayesian deep neural network is... Then the label of the sample will be flipped to We constructed an error-labeled sample that simulates a real error-labeling pattern.

[0073] refer to Figure 4The horizontal axis represents the number of active learning iterations, from round 1 to round 10; the vertical axis represents the ratio, ranging from 0 to 1. The graph contains three curves, corresponding to the noise label interception rate, the review trigger rate, and the correct label mis-trigger rate, respectively. The noise label interception rate is defined as the ratio of the number of genuine mislabeled samples identified and pushed to the review queue by the meta-learning label quality assessment sub-network to the total number of genuine mislabeled samples. This metric started at 0.76 in round 1 and showed a steady upward trend with each iteration, reaching 0.91 by round 10. This reflects the continuously enhanced ability of the meta-learning label quality assessment sub-network to identify genuine mislabeling patterns as the training set size increases and the model becomes more saturated with confusion. The review trigger rate is defined as the ratio of the number of labeled samples pushed to the review queue (including correctly identified incorrect labels and incorrectly judged correct labels) to the total number of labeled samples in the current batch. This indicator decreased from 0.28 in the first round to 0.10 in the tenth round, indicating that as the meta-learning labeling quality assessment sub-network is continuously updated, its confidence in the labeling results is constantly improving, the proportion of incorrect pushes of correct labels is significantly reduced, and the workload of review gradually decreases with iteration. The correct labeling mis-touch rate is defined as the ratio of the number of correctly labeled samples incorrectly pushed to the review queue to the total number of correctly labeled samples. This indicator decreased from 0.12 in the first round to 0.05 in the tenth round, which corroborates the downward trend of the review trigger rate. Together, they indicate that the meta-learning labeling quality assessment sub-network maintains and improves the sensitivity to identify real incorrect labels while reducing invalid reviews. The figure shows the specific values ​​of each indicator in the 10th round. From the combined trend of the continuous increase in noise label interception rate and the continuous decrease in review trigger rate and false trigger rate, it can be clearly observed that the meta-learning label quality evaluation sub-network has achieved adaptive performance improvement as the active learning iteration proceeds. That is, while the label budget is consumed, the quality control capability is also enhanced simultaneously, forming a positive feedback loop of "increased number of labels → model update → more accurate quality evaluation → fewer noise labels entering the training set → more stable model performance".

[0074] This targeted flipping construction method has a fundamental advantage over random flipping: annotation errors in real-world scenarios are almost never uniformly random, but rather highly concentrated on confusion between visually or semantically similar categories. For example, in medical image annotation, annotators might confuse "benign nodule" with "malignant nodule," but would almost never label "fracture" as "pneumonia." Using model confusion pairs to construct erroneous annotation samples in a targeted manner allows the training data distribution of the meta-learning annotation quality assessment sub-network to closely match the real error distribution, significantly improving its sensitivity to identifying real annotation errors.

[0075] Set the quality label of correctly labeled samples to 1 and the quality label of incorrectly labeled samples to 0. Using the concatenated vector of the standardized feature embedding vector and the fusion uncertainty score of each correctly labeled sample, and the concatenated vector of the standardized feature embedding vector and the fusion uncertainty score of each incorrectly labeled sample as input, and the quality label as the supervision signal, perform gradient descent training using cross-entropy loss until the meta-learning annotation quality evaluation sub-network converges. When constructing training samples, it is recommended to maintain a ratio of approximately 1:1 between correctly labeled and incorrectly labeled samples to avoid class imbalance that might cause the meta-learning annotation quality evaluation sub-network to predict all annotation results as correct. If the training set is small (total number of labeled samples less than 200), one corresponding incorrectly labeled sample can be constructed for each labeled sample to make full use of the limited training data.

[0076] In the actual implementation of annotation quality assessment, for each sample of submitted annotation results, the calculated confidence score will be... The data is compared with a preset threshold. The preset threshold is set to 0.6, which means that when the probability of a label being correct, determined by the meta-learning label quality evaluation sub-network, is less than 60%, a secondary manual review mechanism is triggered. Choosing 0.6 instead of a higher threshold (such as 0.8 or 0.9) strikes a balance between review costs and the risk of noisy labels: an excessively high threshold would push a large number of originally correct labels into the review queue, increasing unnecessary manual costs; an excessively low threshold would miss real erroneous labels, failing to effectively protect the quality of training data. Optionally, for scenarios where annotators are experienced and the error rate is historically low, the threshold can be appropriately lowered to 0.5; for scenarios where there are fewer annotators or the overall sample difficulty is high, it can be appropriately increased to 0.7.

[0077] Labeled samples with a confidence score below 0.6 are added to the review queue and re-labeled independently by a second human annotator. The reason for requiring the second annotator to make an independent judgment, rather than informing them of the first annotator's result, is to avoid the anchoring effect—if the second annotator reviews the first annotator's conclusion, their judgment will be influenced by prior information, significantly reducing their independence and the corrective value of the review. The result with the higher confidence score from the two annotations is taken as the final annotation result; labeled samples with a confidence score of 0.6 or higher directly use the initial annotation result and do not need to enter the review process.

[0078] Optionally, for certain application scenarios with extremely high requirements for annotation quality (such as training data annotation for medical diagnostic auxiliary systems), the secondary review mechanism can be extended to a tertiary annotation majority vote: when the annotation results of the first and second annotators are inconsistent, a third annotator is introduced to independently annotate, and the category that appears most frequently in the three annotation results is taken as the final annotation result. This extension method increases labor costs, but it is of great significance for ensuring annotation quality in safety-critical scenarios.

[0079] All confirmed labeled samples in the current batch (including samples directly using the initial labeling results and samples that have been verified) are added to the training set, and then incremental updates are performed on the Bayesian deep neural network. The incremental update adopts a strategy of freezing the feature extraction layer and updating only the classification output layer: the parameters of the feature extraction layer of the Bayesian deep neural network are frozen, and only the classification output layer of the Bayesian deep neural network is updated by performing mini-batch stochastic gradient descent updates on a mini-batch consisting of all confirmed labeled samples in the current batch.

[0080] The rationale for this freezing strategy is reflected on two levels. Firstly, the feature extraction layer has already completed basic feature representation learning through pre-training or a small amount of initial labeled data in the initial stage of the active learning process, possessing the ability to extract meaningful feature embedding vectors in the feature space of the current task. In incremental scenarios where only a few hundred labeled samples are added per round, updating the feature extraction layer, with the update gradient coming from this extremely small subset of samples in the current batch, could very likely lead to overfitting of the feature extraction layer to the distribution of the current batch of samples, damaging its generalization ability across the global distribution, and causing catastrophic forgetting of previously learned feature representations. Secondly, the number of parameters in the classification output layer is much smaller than that in the feature extraction layer. Updating only the classification output layer effectively absorbs the category decision information carried by the newly labeled samples in the current batch while minimizing the range of parameter drift, maintaining the stability of the feature space representation, and ensuring that the calculations based on feature embedding vectors in steps 1 and 2 are comparable across rounds.

[0081] Simultaneously, the meta-learning annotation quality evaluation sub-network is incrementally updated using all confirmed labeled samples in the current batch. The update method is the same as that of the Bayesian deep neural network classification output layer, that is, training data pairs are constructed using the current batch samples according to the directional category flipping method, and mini-batch stochastic gradient descent updates are performed using cross-entropy loss. This allows the meta-learning annotation quality evaluation sub-network to continuously learn the confusion pattern of the current model in the latest feature space and maintain the ability to adaptively adjust the identification of annotation errors as the training set expands.

[0082] After each incremental update, for each category in the training set, the feature embedding vectors of all labeled samples belonging to each category are recalculated by taking the arithmetic mean, and the centroid vector of each category is updated. The necessity of this step stems from the fact that although the parameters of the feature extraction layer are frozen, the update of the classification output layer affects the model's classification of samples. More importantly, the newly added labeled samples to the training set in each round are themselves new members of the feature space. If the centroid vector is not updated in time, the calculation of the DUQ uncertainty score in step 1 in the next round will be based on outdated class centers, leading to a decrease in the accuracy of out-of-distribution detection. The newly added samples may open up previously uncovered areas in the feature space. If the centroid vector is not updated accordingly, these areas will continue to be regarded as out-of-distribution areas and repeatedly selected in the next round, resulting in local concentration of samples rather than global equilibrium. By synchronously refreshing the centroid vector after each incremental update, it can be ensured that the DUQ uncertainty score always reflects the actual coverage of the current training set in the feature space, maintaining the accuracy of uncertainty quantification in step 1 and consistency between rounds.

[0083] Repeat steps 1 through 3 until the termination condition is met. The termination condition is set to one of the following two conditions: First, the annotation budget is exhausted, that is, the total number of labeled samples reaches the pre-set maximum annotation limit; Second, the relative change rate of the average uncertainty score of all unlabeled samples in three consecutive rounds is less than 5%. The latter reflects the uncertainty convergence state of the model: when the overall uncertainty assessment of all unlabeled samples no longer changes significantly after several consecutive rounds of incremental updates, it indicates that the information gain of further annotation to the model is very limited, and the marginal utility of continuing annotation approaches zero. At this time, terminating the iteration is a reasonable and economical choice.

[0084] The relative rate of change is calculated as follows: Let the first... The mean of the fusion uncertainty scores of all unlabeled samples in the round is Then the first The relative rate of change of the wheel ,in For the first The average value of the wheel, Represents absolute value. When , and When both conditions are met, the second type of termination condition is triggered. Meeting the condition for three consecutive rounds, rather than just one, is to avoid a temporary decrease in the mean and accidental termination due to a round that happens to select batches concentrated in a low-uncertainty region. Using a relative rate of change instead of an absolute change ensures that the termination condition is invariant to the magnitude of the fusion uncertainty score; regardless of the overall uncertainty level, the 5% relative change threshold has consistent statistical significance.

[0085] Optionally, in some application scenarios with explicit quantitative requirements for the final performance of the model, a third termination condition can be added in addition to the two termination conditions mentioned above: after each round of incremental updates, the updated Bayesian deep neural network is evaluated using a fixed validation set. Termination is triggered when the classification accuracy on the validation set improves by less than 0.5 percentage points for two consecutive rounds. This condition directly uses the convergence of model performance as the termination criterion, reflecting the actual contribution of annotations to the final model performance more directly than termination conditions based on the mean of uncertainty. It is suitable for industrial deployment scenarios with strict requirements for model accuracy.

[0086] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent data annotation method based on active learning and uncertainty quantification, characterized in that, Includes the following steps: Step 1: For each unlabeled sample in the dataset to be labeled, calculate the BALD uncertainty score and DUQ uncertainty score using a Bayesian deep neural network. Perform intra-batch normalization on the BALD uncertainty score and DUQ uncertainty score respectively, and take the larger value to obtain the fusion uncertainty score of each unlabeled sample. Step 2: Construct a standardized feature matrix based on the feature embedding vectors of all unlabeled samples, calculate the sample covariance matrix corresponding to the standardized feature matrix and perform eigenvalue decomposition, identify anomalous feature values ​​based on the upper bound of the Marchenko-Pastur distribution in random matrix theory, calculate the spectral anomaly response value of each unlabeled sample, and mark unlabeled samples whose spectral anomaly response value exceeds the mean of the spectral anomaly response values ​​of all unlabeled samples as spectrally significant samples. The enhancement score of spectrally significant samples is the sum of the fusion uncertainty score and the spectral anomaly response value. The enhancement score of non-spectrally significant samples is the fusion uncertainty score. Select a preset number of unlabeled samples in descending order of enhancement score to form the current batch of unlabeled sample set. Step 3: Present the current batch of samples to be labeled to human annotators in order of enhancement scores from low to high and collect the labeling results. Calculate the confidence score for each labeling result through the meta-learning labeling quality evaluation sub-network. Trigger a secondary manual review mechanism for labeling results with confidence scores below a preset threshold. Add all confirmed labeled samples to the training set and perform incremental updates on the Bayesian deep neural network. Repeat steps 1 through 3 until the termination condition is met.

2. The method according to claim 1, characterized in that, The calculation process of the BALD uncertainty score in step 1 is as follows: During the inference phase, the Dropout layer of the Bayesian deep neural network is kept active. 50 independent forward propagations are performed on each unlabeled sample to obtain 50 sets of predicted probability outputs for each unlabeled sample. The arithmetic mean of the 50 sets of predicted probability outputs is taken to obtain the average prediction distribution of each unlabeled sample. The information entropy of the average prediction distribution of each unlabeled sample and the arithmetic mean of the information entropy of each of the 50 sets of predicted probability outputs for each unlabeled sample are calculated to obtain the mean conditional entropy of each unlabeled sample. The raw BALD score for each unlabeled sample is obtained by subtracting the mean of the conditional entropy from the information entropy of the average predicted distribution of each unlabeled sample.

3. The method according to claim 2, characterized in that, The calculation process of the DUQ uncertainty score in step 1 is as follows: Each unlabeled sample is passed through the feature extraction layer of a Bayesian deep neural network to output the feature embedding vector of each unlabeled sample; for each category in the training set, the arithmetic mean of the feature embedding vectors of all labeled samples belonging to each category is taken to obtain the centroid vector of each category; the Euclidean distance between the feature embedding vector of each unlabeled sample and the centroid vector of each category in the training set is calculated, and the minimum value of the corresponding Euclidean distances of all categories is taken as the minimum Euclidean distance of each unlabeled sample; the minimum Euclidean distance of each unlabeled sample is divided by the radial basis function bandwidth parameter, and the exponential function value with the natural constant as the base is taken; then the obtained exponential function value is subtracted from 1 to obtain the original DUQ score of each unlabeled sample.

4. The method according to claim 3, characterized in that, The process of intra-batch normalization in step 1 is as follows: For the raw BALD scores of all unlabeled samples, subtract the minimum value among all raw BALD scores of all unlabeled samples from the raw BALD score of each unlabeled sample, and then divide by the difference between the maximum and minimum values ​​of the raw BALD scores of all unlabeled samples to obtain the normalized BALD score of each unlabeled sample. For the raw DUQ scores of all unlabeled samples, subtract the minimum raw DUQ score of all unlabeled samples from the raw DUQ score of each unlabeled sample, and then divide by the difference between the maximum and minimum raw DUQ scores of all unlabeled samples to obtain the normalized DUQ score of each unlabeled sample. Take the larger value between the normalized BALD score and the normalized DUQ score of each unlabeled sample as the fusion uncertainty score of each unlabeled sample.

5. The method according to claim 1, characterized in that, The construction process of the standardized feature matrix and sample covariance matrix in step 2 is as follows: Arrange the feature embedding vectors of all unlabeled samples by rows to construct a feature matrix. The number of rows in the feature matrix is ​​the total number of unlabeled samples, and the number of columns is the dimension of the feature embedding vectors. Calculate the column mean of the feature matrix column by column. Subtract the corresponding column mean from each element in each column of the feature matrix, and then divide each element in each column by the corresponding column standard deviation to obtain the standardized feature matrix. Multiply the transpose of the standardized feature matrix by the standardized feature matrix and divide by the total number of unlabeled samples to obtain the sample covariance matrix. Perform eigenvalue decomposition on the sample covariance matrix to obtain the eigenvalue sequence composed of all eigenvalues ​​and the eigenvector matrix corresponding to each eigenvalue.

6. The method according to claim 5, characterized in that, The process of determining the upper bound of the Marchenko-Pastur distribution and identifying anomalous eigenvalues ​​in step 2 is as follows: calculate the ratio of the number of rows to the number of columns of the standardized eigenvalue matrix, sum the arithmetic square root of the ratio to 1, and then square the result to obtain the theoretical noise upper bound; mark the eigenvalues ​​in the eigenvalue sequence that exceed the theoretical noise upper bound as anomalous eigenvalues.

7. The method according to claim 6, characterized in that, The process of calculating the spectral anomaly response value and determining the current batch of unlabeled samples in step 2 is as follows: For each anomaly feature value, extract the feature vector corresponding to each anomaly feature value from the feature vector matrix, calculate the absolute value of the projection of the standardized feature embedding vector of each unlabeled sample onto the feature vector direction corresponding to each anomaly feature value; take the arithmetic mean of the absolute values ​​of the projection of each unlabeled sample onto the feature vector directions corresponding to all anomaly feature values ​​to obtain the spectral anomaly response value of each unlabeled sample. Unlabeled samples whose spectral anomaly response values ​​exceed the mean of all unlabeled spectral anomaly response values ​​are marked as spectral significant samples; The enhancement score for spectral saliency samples is the sum of the fusion uncertainty score and the spectral anomaly response value of the spectral saliency samples; the enhancement score for non-spectral saliency samples is the fusion uncertainty score of the non-spectral saliency samples; all unlabeled samples are sorted in descending order according to the enhancement score, and the top 300 unlabeled samples are taken to form the current batch of unlabeled sample set.

8. The method according to claim 1, characterized in that, The process of calculating the confidence score through the meta-learning annotation quality assessment sub-network in step 3 is as follows: the standardized feature embedding vector of each submitted annotation result sample is concatenated with the fusion uncertainty score to obtain the concatenated vector of each submitted annotation result sample; the concatenated vector of each submitted annotation result sample is used as the input of the meta-learning annotation quality assessment sub-network, which outputs a 2-dimensional vector. Softmax operation is performed on the 2-dimensional vector, and the value of the first dimension in the 2-dimensional output is the confidence score of the corresponding sample annotation result; annotated samples with a confidence score lower than 0.6 are added to the review queue and re-annotated independently by a second human annotator, and the result with the higher confidence score between the two annotation results is taken as the final annotation result; annotated samples with a confidence score not lower than 0.6 directly use the initial annotation result.

9. The method according to claim 8, characterized in that, The training process of the meta-learning annotation quality assessment sub-network is as follows: Labeled samples are extracted from the training set, and directional class flipping is performed according to the class pairs with the highest confusion predicted by the Bayesian deep neural network on similar feature samples to construct mislabeled samples; the quality label of the correctly labeled samples is set to 1, and the quality label of the mislabeled samples is set to 0; the standardized feature embedding vector of each correctly labeled sample and the concatenated vector of the fusion uncertainty score, and the standardized feature embedding vector of each mislabeled sample and the concatenated vector of the fusion uncertainty score are used as inputs, with the quality label as the supervision signal, and gradient descent training is performed using cross-entropy loss until the meta-learning annotation quality assessment sub-network converges.

10. The method according to claim 1, characterized in that, The incremental update process and termination condition in step 3 are as follows: Freeze the feature extraction layer parameters of the Bayesian deep neural network, and only perform mini-batch stochastic gradient descent update on the classification output layer of the Bayesian deep neural network with a mini-batch consisting of all confirmed labeled samples in the current batch; Simultaneously perform incremental update on the meta-learning annotation quality evaluation sub-network with all confirmed labeled samples in the current batch; After each incremental update, for each category in the training set, recalculate the arithmetic mean of the feature embedding vectors of all labeled samples belonging to each category to update the centroid vector of each category; The termination condition is that the annotation budget is exhausted, or the relative change rate of the mean of the fusion uncertainty score of all unlabeled samples in three consecutive rounds is less than 5%.

Images