AI marking closed-loop method based on active learning and difficult case mining

Abstract

The invention discloses an AI marking closed-loop method based on active learning and difficult case mining, and the method comprises the steps: carrying out the parallel calculation of uncertainty, representativeness and diversity three-dimensional value indexes after training an initial model, and carrying out the dynamic weighting screening of a high-value sample through a closed-loop feedback controller; difficult cases are recognized, DBSCAN clustering is adopted, similar sample expansion is retrieved, and the samples are stored in a dynamic pool; annotations are distributed, intelligent judgment is triggered for dispute samples after consistency verification, and labels are determined by fusing node reputation, model confidence and feature similarity; a mixed loss function is adopted for incremental training, and if the performance does not reach the standard, sampling, difficult examples and training parameters are adjusted in a linkage mode and then retry is conducted; the effectiveness of the difficult cases is evaluated after each round of iteration, the invalid difficult cases are eliminated, redundant clusters are combined, and the pool capacity is adaptively maintained; and the performance of the monitoring model is iteratively optimized until the standard is reached. According to the method, the labeling cost is reduced, and the boundary sample and long tail category recognition capability is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an AI labeling closed-loop method based on active learning and hard example mining. Background Technology

[0002] With the deepening application of artificial intelligence technology in professional fields such as industrial quality inspection, medical diagnosis, and legal text analysis, high-quality labeled data has become a core bottleneck restricting model performance. Traditional supervised learning methods rely on large-scale randomly sampled labeled data, resulting in high labeling costs and low efficiency, especially in long-tail category recognition and boundary sample learning, where they have significant shortcomings.

[0003] Existing active learning techniques primarily employ single strategies such as uncertainty sampling, representative sampling, or diversity sampling to screen unlabeled samples. However, these methods have significant drawbacks: uncertainty sampling tends to concentrate samples at local decision boundaries, ignoring the global data distribution; while representative sampling can cover the distribution, it struggles to accurately locate high-value difficult cases; and although diversity sampling reduces redundancy, it may miss crucial boundary samples. All of these methods use fixed weights to fuse multiple objectives, lacking a dynamic response to changes in model performance. They cannot adaptively adjust sampling strategies based on feedback signals such as boundary sample error rates and distribution bias rates, resulting in limited performance improvements at the same labeling cost.

[0004] In terms of hard example discovery, existing technologies mostly rely on manual screening or simple threshold determination, lacking mechanisms for automatic identification and expansion of hard examples. Traditional methods do not establish a dynamic maintenance mechanism for the hard example pool, resulting in learned but invalid hard examples continuously occupying storage resources, while emerging hard examples cannot be incorporated into the training loop in a timely manner. In addition, the capacity of the hard example pool is mostly statically set, and cannot be adjusted according to model performance to scale gradients, leading to resource waste or insufficient samples.

[0005] Quality control in the annotation process relies on manual review or simple majority voting, and a dynamic reputation evaluation system for annotation nodes has not been established. Existing methods cannot quantify the stability differences of annotators on difficult samples, and the handling of controversial samples lacks a weighted fusion mechanism of model confidence and feature similarity, resulting in large fluctuations in annotation quality and poor model optimization after data retraining.

[0006] Incremental training often employs full parameter updates or fixed-freeze strategies, failing to selectively unfreeze parameter layers based on the contribution of hard examples to misclassification. The weights of hard examples in the hybrid loss function are mostly fixed values, unable to dynamically adjust according to the difference in loss reduction rates between hard examples and ordinary samples. When model performance falls short, existing technologies lack cross-module parameter linkage diagnostic mechanisms; sampling weights, hard example judgment thresholds, and training strategies are adjusted in isolation, making coordinated optimization difficult.

[0007] In summary, existing technologies suffer from core problems such as rigid sampling strategies, lack of closed-loop processing for difficult examples, weak parameter adaptation capabilities, and imperfect quality control mechanisms. These issues lead to high annotation costs, insufficient model learning of boundary samples and long-tail categories, and low data utilization. There is an urgent need for an AI annotation method that integrates dynamic feedback, closed-loop iteration, and cross-module collaboration to achieve the dual goals of reducing annotation costs and improving model performance. Summary of the Invention

[0008] Therefore, this invention provides an AI labeling closed-loop method based on active learning and hard example mining to solve the aforementioned problems in the prior art.

[0009] To achieve the above objectives, this invention provides an AI labeling closed-loop method based on active learning and hard example mining, comprising:

[0010] Step S1: Receive the raw data and extract feature vectors to form an unlabeled sample set, and train the initial model based on the unlabeled sample set;

[0011] Step S2: Calculate the value indicators of unlabeled samples in multiple dimensions according to the initial model, dynamically adjust the weights of each dimension according to historical iteration performance feedback, and select high-value samples as the target set to be labeled after fusion.

[0012] Step S3: Automatically identify difficult examples from the target set to be labeled and model error samples according to preset multidimensional rules, perform unsupervised clustering on the difficult examples and retrieve similar samples to expand them, forming a set of difficult examples and storing it in the dynamic pool.

[0013] Step S4: Distribute the target set to be labeled and the set of difficult examples according to priority, obtain multiple labeling results for the same batch of samples and verify consistency; if the consistency is lower than the threshold, automatically determine the final label by combining the historical data of the labeled nodes, the confidence of the model prediction and the similarity of sample features to construct the labeling database.

[0014] Step S5: Extract new data from the labeled database, construct an incremental training set by combining it with the hard examples in the dynamic pool, lock some parameters of the current model and train a candidate model using a hybrid loss function; if the performance does not meet the preset improvement conditions, adjust the sampling weights, hard example identification parameters and training strategy in a coordinated manner and return to step S2.

[0015] Step S6: After each iteration, the candidate model is used to evaluate the effectiveness of difficult examples in the dynamic pool, eliminate failed difficult examples and merge redundant clusters, so that the capacity of the dynamic pool is adaptively maintained within a preset ratio range.

[0016] Step S7: Monitor the performance of the candidate model. If the target is not met, use it as the base model for the next iteration. After adaptively adjusting the key parameters based on historical iteration information, jump to step S2. Otherwise, terminate and output the target model.

[0017] Furthermore, the process of step S2 includes:

[0018] For each sample in the unlabeled sample set, its predicted probability distribution is obtained through the current iteration base model, and the value indicators of three dimensions—uncertainty, representativeness, and diversity—are calculated in parallel.

[0019] The value indicators of the three dimensions are preprocessed with orthogonality constraints to eliminate information redundancy and obtain each value component.

[0020] The boundary sample error rate, distribution deviation rate, and sampling redundancy rate from the previous iteration are obtained as performance feedback signals and input into the closed-loop feedback controller to generate dynamic weight coefficients for each value component.

[0021] Each value component is weighted and fused according to the dynamic weight coefficient to obtain the comprehensive information value score of the sample, and then sorted in descending order of the score;

[0022] The number of samples for this round is adaptively determined based on the improvement of the macro-average F1 value of the candidate models in the previous round, and the high-value samples with the highest ranking are selected to form the target set to be labeled.

[0023] Furthermore, the process of parallel computing of the value indicators in three orthogonal dimensions includes:

[0024] For each sample in the unlabeled sample set, the original and perturbed predicted probability distributions are obtained by performing two forward inferences through the base model in this round of iteration. The uncertainty score is calculated by fusing the prediction entropy, confidence complement, and mutual information of the two distributions. The similarity between the sample and the global data distribution is evaluated by using adaptive bandwidth kernel density estimation to obtain a representativeness score. The feature cosine distance between the sample and the sample set already sampled in this round is calculated and time decay weighted aggregation is applied to obtain a diversity score.

[0025] Furthermore, the multi-dimensional rules preset in step S3 specifically include:

[0026] For each sample in the target annotation set and the model error samples, the contribution value of hard examples in three dimensions is calculated in parallel:

[0027] Obtain the performance decay rate of the candidate model in the previous round on the hard sample. If the decay rate exceeds the first threshold, increase the weight of the uncertainty dimension. If the F1 value of the long-tail category does not improve, increase the weight of the long-tail dimension. If the accuracy of the boundary sample decreases, increase the weight of the confidence dimension to form the hard sample judgment weight vector for this round.

[0028] The contribution values ​​of the three dimensions of difficult cases are weighted and fused according to the weight vector to generate a comprehensive difficult case index. If the index exceeds the difficult case judgment threshold, the sample is included in the initial set of difficult cases.

[0029] Furthermore, the process of step S4 includes:

[0030] The difficult example set and the target annotation set are distributed to multiple annotation nodes according to priority, with nodes whose historical annotation accuracy is higher than a preset threshold being given priority in the difficult example set;

[0031] For each sample, collect independent annotation results from at least two annotation nodes, calculate the consistency coefficient, and if the coefficient meets the preset standard, it is directly adopted; otherwise, the intelligent adjudication mechanism is triggered.

[0032] During the adjudication process, the dynamic reputation score of each labeled node is obtained. This score integrates the historical accuracy and the labeling stability on hard examples. The prediction confidence of the base model for the sample is obtained and temperature scaling calibration is performed based on the statistical characteristics of the hard example pool. The feature similarity between the sample and the cluster centers of the same type in the hard example pool and its local density weight are calculated.

[0033] The dynamic reputation score, the calibrated confidence score, and the feature similarity are weighted and fused to generate the reliability score of each candidate label. The highest score is selected as the final label. The fusion weight is adaptively adjusted according to the adjudication accuracy of the disputed samples in the previous iteration.

[0034] The final labels are stored in the labeling database, and the dynamic reputation scores of each labeling node are updated in reverse as the basis for the next round of task allocation.

[0035] Furthermore, the process of using the reverse-updated dynamic reputation scores of each standard node as the basis for the next round of task allocation includes:

[0036] The historical accuracy difference of each labeled node on ordinary samples and difficult sample samples is statistically analyzed. When the difference exceeds the preset threshold, it is determined that the node has a difficult sample labeling bias.

[0037] An exponential decay factor is introduced to give higher weight to the recent labeling stability of nodes on difficult sample cases than to their long-term performance;

[0038] The calibrated accuracy and stability are weighted and fused to generate a dynamic reputation score, which is automatically updated after each iteration and serves as the quantitative basis for task allocation in the next round.

[0039] Furthermore, the process of step S5 includes:

[0040] Newly labeled data is extracted from the labeled database and combined with hard example samples in the dynamic hard example pool to construct an incremental training set. The hard example samples are weighted by an exponentially increasing weight according to their residence rounds in the pool.

[0041] In the frozen base model, the gradient changes have converged in the lower-level parameters, and only the top and middle layers whose contribution to the misjudgment of hard examples exceeds a preset threshold are updated.

[0042] The hard example loss weights in the hybrid loss function are adjusted in real time based on the dynamic ratio of the loss decrease rate of hard examples to the loss decrease rate of ordinary examples during training to obtain adaptive loss weight coefficients.

[0043] If the performance of the candidate model does not meet the preset improvement conditions, the step size of the parameter adjustment of each module is automatically calculated based on the performance gap vector, and the aforementioned sampling weights, hard case judgment thresholds and adaptive loss weight coefficients are updated in conjunction with each other before returning to step S2 to be executed again.

[0044] Furthermore, the process of automatically calculating the adjustment step size of each module's parameters based on the performance gap vector, updating the aforementioned sampling weights, hard example determination thresholds, and adaptive loss weight coefficients, and then returning to step S2 for re-execution includes:

[0045] Analyze the performance gap vector of the candidate model. If the recall rate of the boundary sample is lower than the preset recall benchmark and the F1 value of the long-tail category is higher than the preset F1 benchmark, it is determined that the hard case identification is insufficient. At this time, the hard case identification threshold is increased first to adjust the step size, and the sampling weight and adaptive loss weight coefficient of multi-target dynamic sampling are locked.

[0046] If the first-level determination is not valid, and the recall rate of the boundary samples is lower than the aforementioned recall benchmark while the sampling distribution deviation rate is higher than the preset deviation benchmark, then it is determined to be a boundary sample sampling deviation. At this time, the uncertainty weight of multi-target dynamic sampling is increased and the similarity retrieval threshold is reduced simultaneously.

[0047] If the first two levels of judgment are not valid, it is determined that the training strategy is mismatched. At this time, the weight of the hard example loss is increased to the first preset weight value, the parameters of the top two layers of the base model are unfrozen to restore its trainability, and the number of K-value samples for multi-objective dynamic sampling is increased to the preset upper limit of the number of samples. The adjustment operations of the three levels are triggered in sequence, and the performance gap vector is recalculated after each adjustment for iterative diagnosis until the performance improvement conditions are met.

[0048] Furthermore, step S6 includes the following process:

[0049] For each hard sample in the dynamic pool, the prediction confidence is obtained using a candidate model. Combined with the sample's residency rounds in the pool and the long-tail labeling status of its category, a three-level validity determination is performed: if the confidence is higher than the first threshold and it is not a long-tail category, it is marked as a failed hard sample; if it is labeled but the cross-round consistency coefficient is lower than the second threshold, it is marked as a disputed hard sample and its priority is reduced; if it is not labeled and the residency rounds exceed the third threshold, it is marked as an expired hard sample and the marginal sample re-evaluation mechanism is triggered.

[0050] For difficult samples marked as invalid, expired or redundant, perform elimination operation, and at the same time calculate the inter-cluster feature distance and intra-cluster label dispersion between every two clusters. If both are lower than the corresponding threshold, trigger cluster merging operation.

[0051] The upper limit of the dynamic pool capacity is dynamically adjusted based on the performance improvement gradient of the candidate model.

[0052] Furthermore, the step of dynamically adjusting the upper limit of the dynamic pool capacity based on the performance enhancement gradient of the candidate model includes:

[0053] If the gradient exceeds the preset benchmark, the capacity limit is increased to 20% of the total sample size; if the gradient is lower than the preset benchmark, the capacity limit is reduced to 10% of the total sample size; otherwise, the current capacity ratio is maintained.

[0054] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention links multi-dimensional sampling value assessment with a dynamic weight feedback mechanism, automatically adjusting uncertainty, representativeness, and diversity indicators according to the model boundary error rate, ensuring that the collected samples are both difficult to learn and globally representative; the difficult example identification threshold is linked to the model performance decay rate, automatically expanding the range of difficult examples and reducing the strictness of similarity retrieval when boundary recall is insufficient, while the capacity of the difficult example pool adaptively expands and contracts according to the performance improvement gradient, keeping the samples in the pool always at the learning frontier; the reputation score of labeled nodes is calibrated in real time through the difference in the accuracy of difficult example labeling, and the adjudication weight of disputed samples is dynamically optimized based on the historical adjudication accuracy, forming a closed-loop feedback for quality control; incremental training employs parameter locking and a hybrid loss function, with the ratio of the loss weight of difficult examples to that of ordinary samples adjusted in real time according to its decrease rate; when performance is substandard, the sampling weight, difficult example judgment threshold, and loss weight are updated through a three-layer progressive diagnosis linkage, achieving coordinated optimization of parameters in each stage rather than isolated adjustments, thereby significantly improving the boundary sample recall rate and long-tail category recognition ability under the condition of limited labeling costs, achieving a dual gain in labeling efficiency and model performance. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating the AI ​​labeling closed-loop method based on active learning and hard example mining provided by the present invention.

[0056] Figure 2 This is a flowchart illustrating step S2 in the AI ​​labeling closed-loop method based on active learning and hard example mining provided by the present invention. Detailed Implementation

[0057] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0058] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0059] It should be noted that in the description of this invention, the terms "upper", "lower", "left", "right", "inner", "outer", etc., which indicate directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and is not intended to indicate or imply that the device or element must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this invention.

[0060] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0061] Please see Figure 1 As shown, this invention provides an AI labeling closed-loop method based on active learning and hard example mining, comprising:

[0062] Step S1: Receive the raw data and extract feature vectors to form an unlabeled sample set, and train the initial model based on the unlabeled sample set;

[0063] Specifically, text, image, and audio data are received via API / file upload / database synchronization. Text is segmented and stop word filtered; images are scaled to 224×224 and normalized; audio data undergoes MFCC feature extraction and silence removal, with batch processing of 32-128 data entries per run. Text is extracted using BERT-base to obtain 768-dimensional features, images using ResNet-50 to obtain 2048-dimensional features, and audio data using CNN+LSTM to obtain 128-dimensional features. Feature vectors are stored in MySQL, and frequently accessed features are cached in Redis (with a 2-hour expiration time and a 15% memory limit). 5%~10% (100-10000 entries) of seed labeled data is loaded and matched with the corresponding base model (BERT for text, ResNet for images, and CNN+LSTM for audio). Training is performed for 10-20 epochs with a batch size of 32 and a learning rate of 0.001. If there is no improvement in performance on the validation set after 3 epochs, training is stopped early. After training, the model is packaged and stored in a database and managed in a versioned manner.

[0064] Step S2: Calculate the value indicators of unlabeled samples in multiple dimensions according to the initial model, dynamically adjust the weights of each dimension according to historical iteration performance feedback, and select high-value samples as the target set to be labeled after fusion.

[0065] Specifically, such as Figure 2 As shown, the process of step S2 includes:

[0066] Step S21: For each sample in the unlabeled sample set, obtain its predicted probability distribution through the base model of this round of iteration, and calculate the value indicators of uncertainty, representativeness and diversity in parallel.

[0067] Specifically, the process of parallel computing of the value indicators in three orthogonal dimensions includes:

[0068] For each sample in the unlabeled sample set, the original and perturbed predicted probability distributions are obtained by performing two forward inferences through the base model in this round of iteration. The uncertainty score is calculated by fusing the prediction entropy, confidence complement, and mutual information of the two distributions. The similarity between the sample and the global data distribution is evaluated by using adaptive bandwidth kernel density estimation to obtain a representativeness score. The feature cosine distance between the sample and the sample set already sampled in this round is calculated and time decay weighted aggregation is applied to obtain a diversity score.

[0069] Specifically, for each sample in the unlabeled sample set, the system performs two forward inferences: the first obtains the original predicted probability distribution P_original, and the second obtains the robust predicted distribution P_perturbed by adding Gaussian noise perturbation (perturbation standard deviation σ=0.01) to the input feature vector. Based on these two distributions, three sub-indices are calculated:

[0070] Prediction entropy: H = -Σ(p_i * log(p_i)), where p_i is taken from the i-th class probability value of P_original and is used to measure the dispersion of model predictions.

[0071] Confidence complement: C = 1 - max(P_original), which is 1 minus the maximum predicted probability, reflecting the model's degree of uncertainty about the prediction result.

[0072] Mutual information: MI = KL(P_original || P_perturbed), calculates the KL divergence between two distributions, quantifying the model's sensitivity to small perturbations. The three sub-indices are normalized to the [0,1] interval and then fused together with equal weights to form an uncertainty score.

[0073] Specifically, adaptive bandwidth kernel density estimation (KDE) is used to evaluate the similarity between samples and the global data distribution. The bandwidth parameter h is dynamically calculated using the Silverman rule: h = 1.06 * σ * n^(-1 / 5), where σ is the standard deviation of the eigenvectors and n is the number of unlabeled samples. For each sample x, its KDE value with the global distribution is calculated: f(x) = (1 / n) * Σ[K((x-x_i) / h)], where K is the Gaussian kernel function. The representative score is 1 - f(x), ensuring that sparse samples located in low-density regions receive higher scores.

[0074] Specifically, the normalized cosine distance between the current sample feature vector v and all samples in the sample set S sampled in this round is calculated as: d(v, S) = 1 - (1 / |S|) * Σ[(v·s_i) / (||v||*||s_i||)]. A time decay factor exp(-t / 5) is applied to the distance, where t is the difference between the entry time of sample s_i and the current time (to avoid resampling recent samples). The final diversity score is a weighted average distance; the larger the distance, the higher the score.

[0075] Step S22: Perform orthogonality constraint preprocessing on the value indicators of the three dimensions to eliminate information redundancy and obtain each value component;

[0076] Specifically, the calculated three-dimensional value indicators are subjected to Gram-Schmidt orthogonalization to eliminate information redundancy. The uncertainty index vector is used as the baseline u1, and the representative index vector u2' is subtracted from its projection onto u1: u2 = u2' - (u2'·u1) / (u1·u1) * u1. The diversity index vector u3' is then subtracted from its projection onto u1 and u2 respectively. This process is implemented through matrix QR decomposition, with a computational complexity of O(3n), which can be accelerated in parallel on a GPU. After orthogonalization, the value components are linearly independent, ensuring no information overlap during subsequent weighted fusion.

[0077] Step S23: Obtain the boundary sample error rate, distribution deviation rate and sampling redundancy rate of the previous iteration as performance feedback signals, and input them into the closed-loop feedback controller to generate dynamic weight coefficients for each value component.

[0078] Specifically, three quantitative performance metrics from the previous iteration are used as feedback signals to input into the closed-loop controller: Boundary Sample Error Rate: the proportion of misclassifications of the base model on boundary samples, with a threshold of 30%; exceeding this indicates insufficient uncertainty sampling; Distribution Bias Rate: the KL divergence between the class distribution of sampled samples and the distribution of real data, with a threshold of 0.2; exceeding this indicates representative sampling imbalance; Sampling Redundancy Rate: the proportion of new sampled samples with feature similarity greater than 0.9 compared to historical sampled samples, with a threshold of 15%; exceeding this indicates insufficient diversity sampling. These three performance metrics are input into an incremental closed-loop feedback controller (using a simplified version of the PID control law), outputting the dynamic weight coefficients of each value component. The controller formula is:

[0079] w_i(t) = w_i(t-1) + α × (e_i(t) - β × e_i(t-1))

[0080] Where w_i(t) is the weight of the i-th dimension (i∈{uncertainty, representativeness, diversity}) in this round, e_i(t) is the error (actual value - threshold) of the corresponding performance index, α is the integral coefficient (value from 0.1 to 0.3), and β is the differential coefficient (value from 0.5 to 0.8). The initial weights are set equally to 1 / 3, and are automatically adjusted after each iteration to ensure that the sum of the three weights is 1.

[0081] Step S24: Weigh and fuse each value component according to the dynamic weight coefficient to obtain the comprehensive information value score of the sample and sort them in descending order of the score;

[0082] Specifically, the three orthogonalized value components are weighted and fused using dynamic weight coefficients: Score = w1*uncertainty + w2*representativeness + w3*diversity, yielding a comprehensive information value score for each sample. The score calculation employs GPU batch processing, capable of processing 100,000 samples per batch with a computation time of less than 1 second. All samples are sorted in descending order of score; in cases of identical scores, samples with earlier entry times are prioritized to ensure fairness.

[0083] Step S25: Adaptively determine the number of samples for this round based on the improvement of the macro-average F1 value of the candidate models in the previous round, and select the high-value samples with the highest ranking to form the target annotation set.

[0084] Specifically, the sampling quantity K for this round is dynamically determined based on the improvement ΔF1 of the macro-average F1 value of the previous round of candidate models:

[0085] If ΔF1 ≥ 5%, it indicates that the model is in a fast learning phase. Increase the sampling rate to accelerate convergence, and increase the K value by 20% (upper limit is 500).

[0086] If 3% ≤ ΔF1 < 5%, maintain the current sampling efficiency and keep the K value unchanged;

[0087] If ΔF1 < 3%, it indicates that the marginal benefit of the current sampling strategy is diminishing. To save costs, the sampling amount should be reduced, and the K value should be reduced by 20% (lower limit is 100).

[0088] The top-K samples are selected to form the target annotation set. The orthogonalization parameters, dynamic weight coefficients, and K-value adjustment records are written into the parameter configuration library for traceability. After sampling is completed, a notification of the samples to be annotated is sent to the hard case mining module, and sampling metadata (including sample ID, sampling weight, and original value of value component) is sent to the annotation module.

[0089] Step S3: Automatically identify difficult examples from the target set to be labeled and model error samples according to preset multidimensional rules, perform unsupervised clustering on the difficult examples and retrieve similar samples to expand them, forming a set of difficult examples and storing it in the dynamic pool.

[0090] Specifically, the multi-dimensional rules preset in step S3 include:

[0091] For each sample in the target annotation set and the model error samples, the contribution value of hard examples in three dimensions is calculated in parallel.

[0092] Specifically, for each sample in the target annotation set and the inference error samples generated by the base model in this iteration, the system simultaneously calculates the hard example contribution value in three dimensions:

[0093] Uncertainty contribution value: The uncertainty score calculated in step S21 is directly called and normalized to the [0,1] interval, without the need for repeated calculation.

[0094] Long-tail contribution value: Query the proportion of the category to which this sample belongs among the globally unlabeled samples. If the category proportion is <5% (the threshold for determining long-tail categories), a fixed base contribution value of 0.5 is assigned; if the cumulative improvement of the F1 score of this category in the first 3 iterations is <2%, an additional reinforcement contribution value of 0.3 is added to dynamically compensate for the model's insufficient learning of long-tail categories.

[0095] Confidence contribution value: Calculated based on the degree of boundary ambiguity of the model's predicted probability. Let the predicted probability distribution of sample x be p=[p1,p2,...,pc]. After sorting the probability values ​​in descending order, calculate the difference between the two largest probabilities, Δ=max(p)-second(p). If Δ<0.1 (boundary ambiguity threshold), the sample is considered to be near the decision boundary, and a confidence contribution value of 0.8 is assigned; if Δ>0.5, 0.1 is assigned; otherwise, linear interpolation is used.

[0096] Obtain the performance decay rate of the candidate model in the previous round on the hard sample. If the decay rate exceeds the first threshold, increase the weight of the uncertainty dimension. If the F1 value of the long-tail category does not improve, increase the weight of the long-tail dimension. If the accuracy of the boundary sample decreases, increase the weight of the confidence dimension to form the hard sample judgment weight vector for this round.

[0097] Specifically, the performance degradation rate ΔP of the previous round of candidate models on difficult examples is obtained. This degradation rate is defined as: (decline in recall of difficult examples + decrease in F1 score of difficult examples) / 2. ΔP is then input into the incremental feedback controller.

[0098] If ΔP > 15% (first threshold), it indicates that the model's ability to learn difficult examples has diminished, and the weight of the uncertainty dimension will be increased by 0.2.

[0099] If the F1 score of the long-tail category does not improve (the improvement is less than 1%), the weight of the long-tail dimension will be increased by 0.15.

[0100] If the accuracy of the boundary samples decreases (the decrease is greater than 5%), the weight of the confidence dimension will be increased by 0.1.

[0101] The initial weights for the three dimensions are all 1 / 3. After adjustment, normalization is performed so that their sum is 1, forming the weight vector w=[w_unc, w_long, w_conf] for this round of hard case judgment.

[0102] The contribution values ​​of the three dimensions of difficult cases are weighted and fused according to the weight vector to generate a comprehensive difficult case index. If the index exceeds the difficult case judgment threshold, the sample is included in the initial set of difficult cases.

[0103] Specifically, the contribution values ​​of the three difficult examples are weighted and fused using a weight vector: Comprehensive Difficulty Index = w_unc × Uncertainty + w_long × Long Tail + w_conf × Confidence. If the index > 0.6 (the difficulty judgment threshold), the sample is included in the initial difficult example set H. The difficulty judgment threshold of 0.6 is set empirically and can be adjusted within the range of [0.5, 0.7] according to the task complexity.

[0104] Specifically, the DBSCAN density clustering algorithm is used for the initial set of hard examples H. Core parameter settings: Neighborhood radius ε: adaptively calculated based on the standard deviation of the feature vectors of H, ε = 0.3 × median_std, where median_std is the median of the standard deviations of all feature dimensions. This coefficient of 0.3 balances the sparsity of the feature space with the clustering granularity; Minimum number of samples MinPts: set to max(5, sqrt(|H|)) to ensure that even a small set of hard examples can form effective clusters, while avoiding over-clustering. The clustering output consists of k hard example clusters {C1, C2, ..., Ck}, each containing hard example samples with similar features. If a sample does not belong to any cluster, it is marked as a noisy hard example and processed separately.

[0105] Specifically, for each hard cluster Ci, its center vector is calculated: vi = mean({x | x∈Ci}), which is the mean of the feature vectors of all samples within the cluster. Samples with similarity exceeding the similarity retrieval threshold with vi are retrieved from the unlabeled samples in the original database. The similarity metric is cosine similarity, and the threshold is dynamically set as follows: if the purity of the previous clustering (consistency of sample labels within the cluster) > 80%, the threshold is set to 0.85 to strictly filter high-quality similar samples; if the purity < 60%, the threshold is reduced to 0.75 to broaden the retrieval range. The number of samples retrieved for each cluster is min(10×|Ci|, 500) of the cluster's sample size, with an upper limit of 500 to prevent excessive expansion of a single cluster. Similar samples retrieved were included in the difficult example augmentation set E after type-specific data augmentation: Text: synonym replacement (replacement ratio 30%), random insertion / deletion of words (10%), sentence order rearrangement; Image: random rotation ±15°, horizontal flip, brightness adjustment ±20%, Gaussian blur (kernel size 3×3); Speech: speech rate change ±15%, volume scaling ±10dB, addition of white noise (SNR≥20dB).

[0106] Specifically, the expanded set of difficult examples, E, is stored in a dynamic difficult example pool. The storage structure adopts a hierarchical design: the first-level directory is divided into folders according to categories; the second-level directory creates subfolders under each category according to cluster IDs to store the corresponding difficult example samples; the metadata file uses JSON format to record the sample ID, cluster ID, annotation status (0: unannotated, 1: annotated, 2: disputed), residency round, similarity score, and creation timestamp for each difficult example. The dynamic pool capacity is set to an initial capacity of 10% of the total sample size.

[0107] Step S4: Distribute the target set to be labeled and the set of difficult examples according to priority, obtain multiple labeling results for the same batch of samples and verify consistency; if the consistency is lower than the threshold, automatically determine the final label by combining the historical data of the labeled nodes, the confidence of the model prediction and the similarity of sample features to construct the labeling database.

[0108] Specifically, step S4 includes the following process:

[0109] The difficult example set and the target annotation set are distributed to multiple annotation nodes according to priority, with nodes whose historical annotation accuracy is higher than a preset threshold being given priority in the difficult example set;

[0110] Specifically, the system distributes the hard example set and the target annotation set to multiple annotation nodes according to priority. The distribution strategy employs a dual-queue mechanism: A high-priority queue is specifically allocated to samples from the hard example expansion set and is only open to nodes with a historical annotation accuracy higher than 0.9 (a preset threshold). Node eligibility is dynamically evaluated in each iteration; if a node's Kappa coefficient in the previous hard example annotation round is <0.75, its high-priority queue access is suspended for one round. A normal-priority queue allocates samples from the target annotation set, which can be processed by all annotation nodes. For each sample, the system provides the predicted class of the base model in this iteration, the calibrated confidence score, and the hard example label (yes / no). Node parallelism is controlled to process no more than 5 samples simultaneously on each node to prevent task overload. The distribution module uses a Redis priority queue data structure to ensure that hard example samples are retrieved first.

[0111] For each sample, collect independent annotation results from at least two annotation nodes, calculate the consistency coefficient, and if the coefficient meets the preset standard, it is directly adopted; otherwise, the intelligent adjudication mechanism is triggered.

[0112] Specifically, each sample is assigned to at least two annotation nodes (preferably three to support majority voting). After annotation nodes submit labels via a web-based visual interface, the system collects all annotation results for the same batch of samples and calculates the Cohen's Kappa consistency coefficient: Kappa ≥ 0.8: Indicates good consistency, and the majority node vote is directly adopted as the final label. If three nodes annotate and the three labels are mutually different (rare), the system downgrades and triggers intelligent adjudication. Kappa < 0.8: Indicates a disputed sample, and the intelligent adjudication mechanism is automatically triggered. If the proportion of disputed samples exceeds 15% of the total annotations, the system sends a warning to the administrator.

[0113] During the adjudication process, the dynamic reputation score of each labeled node is obtained. This score integrates the historical accuracy and the labeling stability on hard examples. The prediction confidence of the base model for the sample is obtained and temperature scaling calibration is performed based on the statistical characteristics of the hard example pool. The feature similarity between the sample and the cluster centers of the same type in the hard example pool and its local density weight are calculated.

[0114] Specifically, when consistency is insufficient, the system initiates intelligent adjudication through three-layer information fusion:

[0115] First layer: For each node participating in the annotation, calculate its reputation score R in real time.

[0116] R = 0.6 × A_acc + 0.4 × A_stability;

[0117] Here, A_acc represents historical accuracy, the node's accuracy across all historical tasks, with the statistical window being the most recent 500 labeled records (if less than 500, all records are counted). A_stability represents the stability of hard example labeling, calculated as the difference between the node's accuracy on hard examples and its accuracy on ordinary examples. If the difference is >0.15 (a preset deviation threshold), the node is considered to have hard example labeling bias, and its stability score is multiplied by a penalty coefficient of 0.8; otherwise, the stability score is 1.0. To enhance recent performance, an exponential decay factor λ=0.9 is applied to the stability score, giving higher weight to labeling behavior in the most recent 10 rounds than earlier behavior. The reputation score is automatically updated after each iteration and stored in the labeled node profile database, serving as the core quantitative basis for subsequent task allocation.

[0118] The confidence level of the original output of the second-layer base model has an overconfidence bias and needs to be calibrated based on the statistical characteristics of the hard example pool: 200 labeled samples are extracted from the dynamic hard example pool as a calibration set; the temperature parameter T is optimized on the calibration set, with the objective function being the cross-entropy loss: min_T - Σ(log(softmax(z_i / T))), where z_i is the model's logits output; usually T>1 (e.g., 1.2~1.5) to reduce the model's overconfidence in hard examples; the calibrated confidence level P_cal = max(softmax(logits / T)) is used for the decision weight calculation.

[0119] The third layer calculates the cosine similarity S_cos between the feature vector of the disputed sample and the cluster centers of similar samples in the dynamic hard example pool (if there are no similar samples in the hard example pool, the mean of similar samples in the target set to be labeled is used instead). The sample's k=5 nearest neighbors are retrieved in the feature space (k value is adjustable in [3,10]), and the average Euclidean distance d_avg is calculated. The density weight W_density = exp(-d_avg), with lower weights for smaller distances (higher sample density) to avoid excessive influence of high-density samples on the decision.

[0120] The dynamic reputation score, the calibrated confidence score, and the feature similarity are weighted and fused to generate the reliability score of each candidate label. The highest score is selected as the final label. The fusion weight is adaptively adjusted according to the adjudication accuracy of the disputed samples in the previous iteration.

[0121] Specifically, for each candidate label l, its reliability score Score(l) is calculated as follows:

[0122] Score(l) = α×Σ(R_i×I(label_i=l))+β× P_cal×I(model_pred=l)+ γ×S_cos×W_density

[0123] Where I(·) is the indicator function, and α, β, γ are the fusion weights (initial values ​​0.4, 0.3, 0.3), which are adaptively adjusted based on the adjudication accuracy of disputed samples in the previous iteration: if the adjudication accuracy > 0.95, the weights are maintained; if < 0.85, β increases by 0.1, and α and γ each decrease by 0.05. Finally, the label with the highest score is selected as the final label.

[0124] The final labels are stored in the labeling database, and the dynamic reputation scores of each labeling node are updated in reverse as the basis for the next round of task allocation.

[0125] Specifically, the process of using the reverse-updated dynamic reputation scores of each standard node as the basis for the next round of task allocation includes:

[0126] The historical accuracy difference of each labeled node on ordinary samples and difficult sample samples is statistically analyzed. When the difference exceeds the preset threshold, it is determined that the node has a difficult sample labeling bias.

[0127] Specifically, a profile database is established for each labeled node, storing its most recent 500 labeled records in a FIFO queue (if less than 500, all records are stored). Each record includes the sample ID, sample type (normal / hard), label, entry timestamp, and label source (human / intelligent decision-making). Normal sample accuracy A_normal: the percentage of records where the node's label matches the final label among samples labeled "normal"; Hard sample accuracy A_hard: the percentage of records where the node's label matches the final label among samples labeled "hard". Accuracy difference ΔA = A_normal - A_hard. If ΔA > 0.15 (preset difference threshold), the node is considered to have hard sample labeling bias, indicating that the node excels at normal labeling but struggles with complex boundary samples. At this point, the node enters an observation period, and its high-priority queue access is suspended for two iterations.

[0128] An exponential decay factor is introduced to give higher weight to the recent labeling stability of nodes on difficult sample cases than to their long-term performance;

[0129] Specifically, an exponential decay factor λ = 0.9 (typical range [0.85, 0.95]) is introduced to calculate the annotation stability S of nodes on hard examples: S = Σ (λ^t × I_t) / Σ λ^t

[0130] Where t is the time step tracing back from the current round (t=0 represents the current round), and I_t is the indicator function (I_t=1 if the node's hard example label in round t matches the final label, otherwise it is 0). This formula makes the weight contribution of the labeling behavior in the most recent 10 rounds greater than that in the earlier 20 rounds, achieving exponential forgetting of long-term performance and high weight reinforcement of recent performance. Stability calculation is triggered after each iteration, and only the node's behavior on hard example samples is counted; ordinary samples are not included in the calculation.

[0131] The calibrated accuracy and stability are weighted and fused to generate a dynamic reputation score, which is automatically updated after each iteration and serves as the quantitative basis for task allocation in the next round.

[0132] Specifically, the node dynamic reputation score R is generated through weighted fusion: R = 0.6 × A_acc + 0.4 × S

[0133] The overall accuracy, A_acc, represents the total accuracy of the entire sample set, and its calculation method is not limited by sample type. The weighting coefficients of 0.6 and 0.4 are based on empirical settings and can be adjusted within the ranges of [0.5, 0.7] and [0.3, 0.5], reflecting the system's evaluation orientation of prioritizing overall accuracy while also considering the stability of difficult examples. If a node is determined to have a difficult example labeling bias (ΔA > 0.15), its reputation score undergoes a punitive decay: R_penalty = R × 0.8 (the penalty coefficient 0.8 can be adjusted within the range of [0.7, 0.9] according to the task's rigor). The reputation score R is automatically written to the "Reputation Score Table" in the node profile database after each iteration, with an update latency of <100ms, ensuring that the latest value is read during the next task distribution. The top 30% of nodes with the highest R values ​​are prioritized for entry into the high-priority queue. Nodes with R values ​​below 0.6 are only assigned ordinary samples, and nodes with R values ​​below 0.5 have their task allocation suspended and trigger a retesting process.

[0134] Step S5: Extract new data from the labeled database, construct an incremental training set by combining it with the hard examples in the dynamic pool, lock some parameters of the current model and train a candidate model using a hybrid loss function; if the performance does not meet the preset improvement conditions, adjust the sampling weights, hard example identification parameters and training strategy in a coordinated manner and return to step S2.

[0135] Specifically, step S5 includes the following process:

[0136] Newly labeled data is extracted from the labeled database and combined with hard example samples in the dynamic hard example pool to construct an incremental training set. The hard example samples are weighted by an exponentially increasing weight according to their residence rounds in the pool.

[0137] Specifically, newly labeled data is extracted from the annotation database, with the following selection criteria: the annotation time is later than the start time of the previous iteration, the annotation confidence is ≥0.6 (the low-quality annotation filtering threshold), and the sample ID has not appeared in the incremental training set. The amount of newly extracted data is denoted as N_new. Simultaneously, hard examples are extracted from the dynamic hard example pool, with the following extraction rules: only hard examples with an annotation status of "annotated" are extracted; an exponentially increasing weight w_hard= λ_round^(t-1) is assigned based on the number of times the sample resides in the pool (t), where λ_round=1.2 (residence weight coefficient, typical range [1.1, 1.3]), and the number of rounds (t) is counted from the first time the sample enters the pool. For example, the weight of a hard example residing in round 1 is 1.0, in round 2 it is 1.2, in round 3 it is 1.44, and so on.

[0138] The weighted sampling process employs a stratified sampling strategy: when constructing the incremental training set, the proportion of hard examples is mandated to be no less than 20% (if the hard example pool is insufficient, it will be randomly supplemented from the new data), and the proportion of long-tail category samples is no less than 10%. The final training set size N_total = min(N_new + N_hard, 2000), with an upper limit of 2000 to prevent excessively long training times. After sampling, the data is divided into a training subset and a validation subset in an 8:2 ratio.

[0139] In the frozen base model, the gradient changes have converged in the lower-level parameters, and only the top and middle layers whose contribution to the misjudgment of hard examples exceeds a preset threshold are updated.

[0140] Specifically, for the first L layers of the model (L=8 for text tasks, L=10 for image tasks, and L=3 for speech tasks), if the fluctuation range of the L2 norm of the gradient in the most recent 3 iterations is <1e-4 (convergence threshold), then the weights of that layer are frozen, and backpropagation is skipped during training. The value of L is set according to the model depth, and the lower layer of feature extraction is usually frozen to preserve general knowledge.

[0141] Selective updates for top and intermediate layers: The contribution of the remaining layers to misclassifications of difficult examples is calculated. Specifically, on the validation subset, the gradient contribution of each layer's activation value to the misclassification of difficult examples is calculated. If the gradient contribution of a certain layer exceeds 15% (the contribution threshold), it is marked as a trainable layer; otherwise, the weights of that layer are frozen. The contribution threshold of 15% can be adjusted within the range of [10%, 20%]. A smaller value results in more trainable layers and a stronger model fitting ability, but also increases computational cost.

[0142] The hard example loss weights in the hybrid loss function are adjusted in real time based on the dynamic ratio of the loss decrease rate of hard examples to the loss decrease rate of ordinary examples during training to obtain adaptive loss weight coefficients.

[0143] Specifically, training uses a hybrid loss function: Loss = Loss_base + w_hard × Loss_hard

[0144] Where Loss_base represents the base loss, i.e., the standard cross-entropy loss, which is the average loss value calculated for all samples. Loss_hard represents the hard-case weighted loss, which calculates the cross-entropy only for hard-case samples. The loss weight w_hard is an adaptive weight coefficient, not a fixed value. The average loss decrease rate of hard-case samples is calculated as ΔL_hard = (Loss_hard(e-1) - Loss_hard(e)) / Loss_hard(e-1); the average loss decrease rate of normal samples is calculated as ΔL_base = (Loss_base(e-1) - Loss_base(e)) / Loss_base(e-1); if ΔL_hard / ΔL_base < 0.8 (hard-case learning lag threshold), it means that the loss decreases more slowly than that of normal samples, and w_hard is increased by 0.3 (weight adjustment step size, range [0.2, 0.5]); if the ratio > 1.2, it is decreased by 0.2. The initial value of w_hard is set to 2.0, and the dynamic adjustment range is limited to [1.0, 5.0] to prevent weight explosion.

[0145] If the performance of the candidate model does not meet the preset improvement conditions, the step size of the parameter adjustment of each module is automatically calculated based on the performance gap vector, and the aforementioned sampling weights, hard case judgment thresholds and adaptive loss weight coefficients are updated in conjunction with each other before returning to step S2 to be executed again.

[0146] Specifically, the process of automatically calculating the adjustment step size of each module's parameters based on the performance gap vector, updating the aforementioned sampling weights, hard example determination thresholds, and adaptive loss weight coefficients, and then returning to step S2 for re-execution includes:

[0147] Analyze the performance gap vector of the candidate model. If the recall rate of the boundary sample is lower than the preset recall benchmark and the F1 value of the long-tail category is higher than the preset F1 benchmark, it is determined that the hard case identification is insufficient. At this time, the hard case identification threshold is increased first to adjust the step size, and the sampling weight and adaptive loss weight coefficient of multi-target dynamic sampling are locked.

[0148] Specifically, if the margin of error recall gap is greater than 0.15 and the long-tail F1 score has exceeded the baseline, it is determined that there is insufficient identification of hard examples. In this case, the hard example identification threshold is lowered by 0.05 (from 0.6 to 0.55) to expand the range of hard example identification, while keeping the sampling weight and loss weight unchanged, focusing on solving the identification problem. After adjustment, return to step S2. If the margin of error recall rate increases by 5%, it is determined to be effective; otherwise, proceed to the next level.

[0149] If the first-level determination is not valid, and the recall rate of the boundary samples is lower than the aforementioned recall benchmark while the sampling distribution deviation rate is higher than the preset deviation benchmark, then it is determined to be a boundary sample sampling deviation. At this time, the uncertainty weight of multi-target dynamic sampling is increased and the similarity retrieval threshold is reduced simultaneously.

[0150] Specifically, if the first-level conditions are not met, and the boundary recall gap is still greater than 0.1 while the distribution bias rate exceeds 0.15, it is determined to be sampling bias. At this time, the uncertainty weight is increased by 0.15, and the similarity retrieval threshold is reduced to 0.78 to expand the retrieval range. After adjustment, return to step S2. If the distribution bias rate decreases by 0.05 and the boundary recall increases by 3%, it is determined to be effective; otherwise, proceed to the third level.

[0151] If the first two levels of judgment are not valid, it is determined that the training strategy is mismatched. At this time, the weight of the hard example loss is increased to the first preset weight value, the parameters of the top two layers of the base model are unfrozen to restore its trainability, and the number of K-value samples for multi-objective dynamic sampling is increased to the preset upper limit of the number of samples. The adjustment operations of the three levels are triggered in sequence, and the performance gap vector is recalculated after each adjustment for iterative diagnosis until the performance improvement conditions are met.

[0152] Specifically, if the first two levels do not meet the requirements, it is determined that the training strategy is mismatched. At this time, three layers of adjustment are performed sequentially:

[0153] First layer: Increase the lower limit of the loss weight for difficult examples to 3.0 to strengthen the learning intensity of difficult examples;

[0154] Second layer: Unfreeze the parameters of the top two layers of the base model to restore its trainability. The learning rate of the unfrozen layer is set to 10% of the global learning rate.

[0155] The third layer: Increase the upper limit of the number of samples to 300 to expand the scale of labeling.

[0156] After each layer adjustment, the process returns to step S2 for retraining and evaluation. If the total performance gap decreases by 20%, the process is considered effective and terminates; otherwise, it continues to the next layer. If there is still no improvement after three layers of adjustments, the termination mechanism is triggered. The maximum number of retry rounds is three progressive adjustments per iteration; exceeding this limit will force termination. If the candidate model's performance decreases by more than 1% after adjustment, the parameters are immediately rolled back to the previous configuration. When the cumulative labeled sample size reaches 50% of the total sample size, the iteration terminates regardless of whether the performance target has been met.

[0157] Step S6: After each iteration, the candidate model is used to evaluate the effectiveness of difficult examples in the dynamic pool, eliminate failed difficult examples and merge redundant clusters, so that the capacity of the dynamic pool is adaptively maintained within a preset ratio range.

[0158] Specifically, step S6 includes the following process:

[0159] For each hard sample in the dynamic pool, the prediction confidence is obtained using a candidate model. Combined with the sample's residency rounds in the pool and the long-tail labeling status of its category, a three-level validity determination is performed: if the confidence is higher than the first threshold and it is not a long-tail category, it is marked as a failed hard sample; if it is labeled but the cross-round consistency coefficient is lower than the second threshold, it is marked as a disputed hard sample and its priority is reduced; if it is not labeled and the residency rounds exceed the third threshold, it is marked as an expired hard sample and the marginal sample re-evaluation mechanism is triggered.

[0160] Specifically, after each iteration, the system automatically traverses all samples in the dynamic hard example pool and performs a three-level validity determination using the prediction results of the candidate models in this round and the sample metadata:

[0161] The candidate model infers for each hard case sample and obtains its prediction confidence P. If P > the first threshold of 0.8 (high confidence threshold) and the sample belongs to a category with a proportion ≥ 5% (non-long-tail category), it is marked as a failed hard case. The judgment logic is: the model has grasped the features of the sample and it does not belong to a long-tail scarce category; continuing to retain it would waste storage and labeling resources. The confidence threshold of 0.8 can be adjusted in the range of [0.75, 0.85], with higher values ​​indicating stricter judgment.

[0162] For labeled difficult examples, calculate their cross-cycle consistency coefficient: calculate the Cohen's Kappa coefficient of all labeled results for that example in the most recent 3 iterations. If Kappa < the second threshold of 0.6, it indicates that the labeling results fluctuate drastically across different iterations, and it is marked as a controversial difficult example, with its sampling priority reduced by 50% (subsequent sampling weights multiplied by 0.5). This threshold of 0.6 is lower than the conventional standard of 0.8, reflecting a more lenient consistency requirement across time dimensions, and can be adjusted within the range of [0.5, 0.7].

[0163] For unlabeled hard examples, check their residency period t. If t > the third threshold of 5 rounds (the upper limit of the residency period), they are marked as expired hard examples, and a marginal sample re-evaluation mechanism is triggered: this mechanism inputs the feature vector of the expired hard example into the candidate model of this round. If the model's prediction entropy for it is >1.0 (highly uncertain), it is reactivated and its priority is increased to the high-priority queue; if the prediction entropy is <0.5, it is confirmed as an invalid hard example and directly eliminated. The third threshold of 5 rounds reflects that hard example samples should complete the labeling and training loop within 5 iterations, and can be adjusted within the range of [3,7] rounds according to the availability of labeling manpower.

[0164] For difficult samples marked as invalid, expired or redundant, perform elimination operation, and at the same time calculate the inter-cluster feature distance and intra-cluster label dispersion between every two clusters. If both are lower than the corresponding threshold, trigger cluster merging operation.

[0165] Specifically, lazy elimination is performed on hard examples marked as invalid, expired, or redundant: instead of immediate physical deletion, their labeling status is set to "inactive" and they are removed from the candidate pool. Batch cleanup occurs when the pool capacity exceeds the limit. The cosine similarity of features between each pair of hard example samples is calculated. If the similarity > 0.95 (redundancy threshold), it is marked as redundant, and only the sample with the smaller residency round is retained. For any two clusters Ci and Cj, the Euclidean distance d_center = ||vi - vj|| between their center vectors is calculated. If d_center < the first merging threshold of 0.3 (center distance threshold), the clusters are considered to have similar features. The class distribution of the labeled samples is statistically analyzed, and the dispersion D = 1 - max (class proportion) is calculated. If D < the second merging threshold of 0.2 (dispersion threshold), the label consistency within the cluster is considered good. When both d_center < 0.3 and D < 0.2 are satisfied, the two clusters are merged, the new cluster ID is taken as the smaller value, and the center vector is recalculated as a weighted average, with the weight being the number of samples within the cluster. After merging, update the metadata of the difficult instance pool to ensure the continuity of the cluster ID mapping relationship.

[0166] The upper limit of the dynamic pool capacity is dynamically adjusted based on the performance improvement gradient of the candidate model.

[0167] Specifically, the step of dynamically adjusting the upper limit of the dynamic pool capacity based on the performance improvement gradient of the candidate model includes:

[0168] If the gradient exceeds the preset benchmark, the capacity limit is increased to 20% of the total sample size; if the gradient is lower than the preset benchmark, the capacity limit is reduced to 10% of the total sample size; otherwise, the current capacity ratio is maintained.

[0169] Specifically, when the number of samples in the pool exceeds the upper limit, invalid and expired difficult cases are eliminated in batches according to the number of difficult case retention rounds from longest to shortest, with the number of cases eliminated each time being 1.5 times the excess amount to leave buffer space.

[0170] Step S7: Monitor the performance of the candidate model. If the target is not met, use it as the base model for the next iteration. After adaptively adjusting the key parameters based on historical iteration information, jump to step S2. Otherwise, terminate and output the target model.

[0171] Specifically, after each iteration, the system evaluates the core performance metrics of the candidate model on an independent test set (10% of the total samples are reserved and not used for training and sampling). The evaluation is performed asynchronously during GPU idle periods to avoid blocking the main process. Monitoring is conducted once per round, with evaluation time controlled within 5 minutes (for 100,000 test samples). Monitoring metrics include: Macro-average F1 score: the arithmetic mean of F1 scores across all categories, reflecting the overall model performance; Long-tail category F1 score: the average F1 score of categories with a sample percentage <5%, reflecting the model's learning ability for scarce categories; Boundary sample recall rate: the proportion of correctly recalled samples in the hard example pool, reflecting the model's mastery of ambiguous boundaries; Cumulative labeling cost: the percentage of labeled samples out of the total sample size, used for cost constraints.

[0172] Specifically, the system sets three levels of termination conditions, and iteration stops when any one of them is met:

[0173] Level 1 (Performance Meets Standard): The candidate model's macro-average F1 score is ≥ the target performance value of 0.92 (preset target), and the improvement rate for two consecutive iterations is <1%. The target value of 0.92 is set according to the task difficulty; it can be increased to 0.95 for simple classification tasks and reduced to 0.88 for complex multi-label tasks.

[0174] Level 2 (Iteration Limit Exceeded): The current iteration rounds have reached the maximum of 50 rounds. This limit prevents infinite loops and ensures system timeliness. For tasks with tens of millions of samples, the limit can be relaxed to 100 rounds; for tasks with hundreds of thousands of samples, the limit is tightened to 30 rounds.

[0175] Level 3 (Cost Exceeding Limit): Cumulative labeled sample size ≥ 50% of total sample size. This limit ensures that labeling costs are controllable; it can be reduced to 30% for cost-sensitive scenarios and relaxed to 70% for accuracy-priority scenarios.

[0176] If the termination condition is not triggered, the system will use the candidate model as the base model for the next iteration and adaptively adjust key parameters based on historical iteration information: if the recall rate of the boundary samples in the previous round improves by <3%, the uncertainty weight of multi-objective dynamic sampling will be increased by 0.1 (maximum 0.5); if the sampling redundancy rate is >15%, the diversity weight will be increased by 0.1 (maximum 0.5). If the effective utilization rate of the hard example pool (number of hard examples participating in training / total number of samples in the pool) is <70%, the hard example judgment threshold will be decreased by 0.05 (lower limit 0.5) to expand the hard example set; if the pool size growth rate is >20% / round, the threshold will be increased by 0.03 (maximum 0.7) to control expansion. If the loss rate of hard example samples decreases more than 20% slower than that of ordinary samples, the adaptive loss weight coefficient w_hard will be increased by 0.5 (maximum 5.0); conversely, if hard examples are overfitted (hard example accuracy > ordinary sample accuracy by 5%), w_hard will be decreased by 0.3 (lower limit 1.0). If the performance improvement is <2% for 3 consecutive iterations, the training learning rate of the base model for the next iteration will be decayed to 50% of the current rate. The parameter adjustment amount is generated by an incremental closed-loop controller. The controller input is the performance gap vector G, and the output is the parameter adjustment command. The adjustment step size is adaptively optimized based on the feedback of the previous adjustment effect (if the adjustment is effective, the step size is increased by 10%; otherwise, it is decreased by 15%).

[0177] Specifically, this invention links multi-dimensional sampling value assessment with a dynamic weight feedback mechanism, enabling uncertainty, representativeness, and diversity indicators to automatically adjust with the model's boundary error rate, ensuring that the collected samples are both difficult to learn and globally representative. The difficult example identification threshold is linked to the model performance decay rate; when boundary recall is insufficient, the range of difficult examples is automatically expanded and the similarity retrieval strictness is reduced. At the same time, the capacity of the difficult example pool adaptively expands and contracts according to the performance improvement gradient, keeping the samples in the pool always at the learning frontier. The reputation score of the labeled nodes is calibrated in real time through the difference in the accuracy of difficult example labeling, and the adjudication weight of disputed samples is dynamically optimized based on the historical adjudication accuracy, forming a closed-loop feedback for quality control. Incremental training adopts parameter locking and a hybrid loss function, and the ratio of the loss weight of difficult examples to that of ordinary samples is adjusted in real time according to its decline rate. When performance is not up to standard, the sampling weight, difficult example judgment threshold, and loss weight are updated in a three-layer progressive diagnosis linkage, realizing the coordinated optimization of parameters in each link rather than isolated adjustment. Thus, under the condition of limited labeling costs, the recall rate of boundary samples and the ability to identify long-tail categories are significantly improved, achieving a dual gain of labeling efficiency and model performance.

[0178] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

[0179] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A closed-loop AI labeling method based on active learning and hard example mining, characterized in that, include: Step S1: Receive the raw data and extract feature vectors to form an unlabeled sample set, and train the initial model based on the unlabeled sample set; Step S2: Calculate the value indicators of unlabeled samples in multiple dimensions according to the initial model, dynamically adjust the weights of each dimension according to historical iteration performance feedback, and select high-value samples as the target set to be labeled after fusion. Step S3: Automatically identify difficult examples from the target set to be labeled and model error samples according to preset multidimensional rules, perform unsupervised clustering on the difficult examples and retrieve similar samples to expand them, forming a set of difficult examples and storing it in the dynamic pool. Step S4: Distribute the target set to be labeled and the set of difficult examples according to priority, obtain multiple labeling results for the same batch of samples and verify consistency; If the consistency is below the threshold, the final label will be automatically determined by combining the historical data of the labeled nodes, the confidence of the model prediction, and the similarity of the sample features to build the label database; Step S5: Extract new data from the labeled database, construct an incremental training set by combining it with the hard examples in the dynamic pool, lock some parameters of the current model and train a candidate model using a hybrid loss function; if the performance does not meet the preset improvement conditions, adjust the sampling weights, hard example identification parameters and training strategy in a coordinated manner and return to step S2. Step S6: After each iteration, the candidate model is used to evaluate the effectiveness of difficult examples in the dynamic pool, eliminate failed difficult examples and merge redundant clusters, so that the capacity of the dynamic pool is adaptively maintained within a preset ratio range. Step S7: Monitor the performance of the candidate model. If the target is not met, use it as the base model for the next iteration. After adaptively adjusting the key parameters based on historical iteration information, jump to step S2. Otherwise, terminate and output the target model.

2. The AI ​​labeling closed-loop method based on active learning and hard example mining according to claim 1, characterized in that, The process of step S2 includes: For each sample in the unlabeled sample set, its predicted probability distribution is obtained through the current iteration base model, and the value indicators of three dimensions—uncertainty, representativeness, and diversity—are calculated in parallel. The value indicators of the three dimensions are preprocessed with orthogonality constraints to eliminate information redundancy and obtain each value component. The boundary sample error rate, distribution deviation rate, and sampling redundancy rate from the previous iteration are obtained as performance feedback signals and input into the closed-loop feedback controller to generate dynamic weight coefficients for each value component. Each value component is weighted and fused according to the dynamic weight coefficient to obtain the comprehensive information value score of the sample, and then sorted in descending order of the score; The number of samples for this round is adaptively determined based on the improvement of the macro-average F1 value of the candidate models in the previous round, and the high-value samples with the highest ranking are selected to form the target set to be labeled.

3. The AI ​​labeling closed-loop method based on active learning and hard example mining according to claim 2, characterized in that, The process of parallel computing of the value indicators in three orthogonal dimensions includes: For each sample in the unlabeled sample set, the original and perturbed predicted probability distributions are obtained by performing two forward inferences through the base model in this round of iteration. The uncertainty score is calculated by fusing the prediction entropy, confidence complement, and mutual information of the two distributions. The similarity between the sample and the global data distribution is evaluated by using adaptive bandwidth kernel density estimation to obtain a representativeness score. The feature cosine distance between the sample and the sample set already sampled in this round is calculated and time decay weighted aggregation is applied to obtain a diversity score.

4. The AI ​​labeling closed-loop method based on active learning and hard example mining according to claim 3, characterized in that, The multi-dimensional rules preset in step S3 specifically include: For each sample in the target annotation set and the model error samples, the contribution value of hard examples in three dimensions is calculated in parallel: Obtain the performance decay rate of the candidate model in the previous round on the hard sample. If the decay rate exceeds the first threshold, increase the weight of the uncertainty dimension. If the F1 value of the long-tail category does not improve, increase the weight of the long-tail dimension. If the accuracy of the boundary sample decreases, increase the weight of the confidence dimension to form the hard sample judgment weight vector for this round. The contribution values ​​of the three dimensions of difficult cases are weighted and fused according to the weight vector to generate a comprehensive difficult case index. If the index exceeds the difficult case judgment threshold, the sample is included in the initial set of difficult cases.

5. The AI ​​labeling closed-loop method based on active learning and hard example mining according to claim 4, characterized in that, The process of step S4 includes: The difficult example set and the target annotation set are distributed to multiple annotation nodes according to priority, with nodes whose historical annotation accuracy is higher than a preset threshold being given priority in the difficult example set; For each sample, collect independent annotation results from at least two annotation nodes, calculate the consistency coefficient, and if the coefficient reaches the preset standard, it is directly adopted; otherwise, the intelligent adjudication mechanism is triggered. During the adjudication process, the dynamic reputation score of each labeled node is obtained. This score integrates the historical accuracy and the labeling stability on hard examples. The prediction confidence of the base model for the sample is obtained and temperature scaling calibration is performed based on the statistical characteristics of the hard example pool. The feature similarity between the sample and the cluster centers of the same type in the hard example pool and its local density weight are calculated. The dynamic reputation score, the calibrated confidence score, and the feature similarity are weighted and fused to generate the reliability score of each candidate label. The highest score is selected as the final label. The fusion weight is adaptively adjusted according to the adjudication accuracy of the disputed samples in the previous iteration. The final labels are stored in the labeling database, and the dynamic reputation scores of each labeling node are updated in reverse as the basis for the next round of task allocation.

6. The AI ​​labeling closed-loop method based on active learning and hard example mining according to claim 5, characterized in that, The process of using the reverse-updated dynamic reputation scores of each standard node as the basis for the next round of task allocation includes: The historical accuracy difference of each labeled node on ordinary samples and difficult sample samples is statistically analyzed. When the difference exceeds a preset threshold, it is determined that the node has a difficult sample labeling bias. An exponential decay factor is introduced to give higher weight to the recent labeling stability of nodes on difficult sample cases than to their long-term performance; The calibrated accuracy and stability are weighted and fused to generate a dynamic reputation score, which is automatically updated after each iteration and serves as the quantitative basis for task allocation in the next round.

7. The AI ​​labeling closed-loop method based on active learning and hard example mining according to claim 6, characterized in that, The process of step S5 includes: Newly labeled data is extracted from the labeled database and combined with hard example samples in the dynamic hard example pool to construct an incremental training set. The hard example samples are weighted by an exponentially increasing weight according to their residence rounds in the pool. In the frozen base model, the gradient changes have converged in the lower-level parameters, and only the top and middle layers whose contribution to the misjudgment of hard examples exceeds a preset threshold are updated. The hard example loss weights in the hybrid loss function are adjusted in real time based on the dynamic ratio of the loss decrease rate of hard examples to the loss decrease rate of ordinary examples during training to obtain adaptive loss weight coefficients. If the performance of the candidate model does not meet the preset improvement conditions, the step size of the parameter adjustment of each module is automatically calculated based on the performance gap vector, and the aforementioned sampling weights, hard case judgment thresholds and adaptive loss weight coefficients are updated in conjunction with each other before returning to step S2 to be executed again.

8. The AI ​​labeling closed-loop method based on active learning and hard example mining according to claim 7, characterized in that, The process of automatically calculating the adjustment step size of each module parameter based on the performance gap vector, updating the aforementioned sampling weights, hard example determination thresholds, and adaptive loss weight coefficients, and then returning to step S2 for re-execution includes: Analyze the performance gap vector of the candidate model. If the recall rate of the boundary sample is lower than the preset recall benchmark and the F1 value of the long-tail category is higher than the preset F1 benchmark, it is determined that the hard case identification is insufficient. At this time, the hard case identification threshold is increased first to adjust the step size, and the sampling weight and adaptive loss weight coefficient of multi-target dynamic sampling are locked. If the first-level determination is not valid, and the recall rate of the boundary samples is lower than the aforementioned recall benchmark while the sampling distribution deviation rate is higher than the preset deviation benchmark, then it is determined to be a boundary sample sampling deviation. At this time, the uncertainty weight of multi-target dynamic sampling is increased and the similarity retrieval threshold is reduced simultaneously. If the first two levels of judgment are not valid, it is determined that the training strategy is mismatched. At this time, the weight of the hard example loss is increased to the first preset weight value, the parameters of the top two layers of the base model are unfrozen to restore its trainability, and the number of K-value samples for multi-objective dynamic sampling is increased to the preset upper limit of the number of samples. The adjustment operations of the three levels are triggered in sequence, and the performance gap vector is recalculated after each adjustment for iterative diagnosis until the performance improvement conditions are met.

9. The AI ​​labeling closed-loop method based on active learning and hard example mining according to claim 8, characterized in that, The process of step S6 includes: For each hard sample in the dynamic pool, the prediction confidence is obtained using a candidate model. Combined with the sample's residency rounds in the pool and the long-tail labeling status of its category, a three-level validity determination is performed: if the confidence is higher than the first threshold and it is not a long-tail category, it is marked as a failed hard sample; if it is labeled but the cross-round consistency coefficient is lower than the second threshold, it is marked as a disputed hard sample and its priority is reduced; if it is not labeled and the residency rounds exceed the third threshold, it is marked as an expired hard sample and the marginal sample re-evaluation mechanism is triggered. For difficult samples marked as invalid, expired or redundant, perform elimination operation, and at the same time calculate the inter-cluster feature distance and intra-cluster label dispersion between every two clusters. If both are lower than the corresponding threshold, trigger cluster merging operation. The upper limit of the dynamic pool capacity is dynamically adjusted based on the performance improvement gradient of the candidate model.

10. The AI ​​labeling closed-loop method based on active learning and hard example mining according to claim 9, characterized in that, The step of dynamically adjusting the upper limit of the dynamic pool capacity based on the performance improvement gradient of the candidate model includes: If the gradient exceeds the preset benchmark, the capacity limit is increased to 20% of the total sample size; if the gradient is lower than the preset benchmark, the capacity limit is reduced to 10% of the total sample size; otherwise, the current capacity ratio is maintained.

Images