A multi-modal data active labeling method based on incremental learning

Abstract

The application discloses a kind of multimodal data active annotation methods;Belong to intelligent annotation method technical field;Its technical points include the following steps: (S1) cold start;(S2) intelligent screening;(S3) difficult mining;(S4) anti-forgetting incremental training: start a new round of model training, training set integrates the anchor sample in the real label and difficult sample pool that are newly added, and through the compound knowledge distillation constraint in loss function and the elastic weight consolidation regular term, construct the anti-forgetting learning mechanism for continuous annotation scene;(S5) closed loop optimization: monitor the memory intensity attenuation of sample in difficult sample pool and artificial correction mode, dynamically adjust the composition and training weight of sample in pool, and feedback optimization signal to the screening threshold of S2 and the training strategy of S4, form adaptive annotation-training closed loop;The application aims to provide a kind of efficient, high-quality and sustainable incremental learning based multimodal data active annotation method;For the active annotation of data.

Description

Technical Field

[0001] This invention relates to an annotation method, and more specifically, to a multimodal data active annotation method based on incremental learning. Background Technology

[0002] With the deepening application of artificial intelligence in fields such as fabric identification, autonomous driving, smart healthcare, and content moderation, the demand for large-scale, high-quality multimodal labeled data (such as images, videos, text, and 3D point clouds) is exploding. Traditional manual annotation methods are time-consuming, labor-intensive, and costly, and have become a key bottleneck restricting the performance improvement of AI models.

[0003] To reduce annotation costs, active learning technology is widely used. Its core idea is to automatically select the "most valuable" unlabeled samples through the model and hand them over to humans for annotation, aiming to obtain a higher-performing model with less annotation cost. However, existing active learning-based annotation systems face three core problems that urgently need to be solved in practical applications, especially in multimodal, long-cycle, and continuously iterative annotation scenarios:

[0004] (1) It is difficult to balance efficiency and quality in the cold start stage: There is a lack of effective models in the initial stage of the system. The traditional approach is to randomly or empirically select a batch of samples for labeling. This blindness leads to unstable quality of the initial model. Either the effective features cannot be learned due to insufficient sample size, or valuable labeling resources are wasted due to sample redundancy, which directly affects the starting point and efficiency of the subsequent active learning cycle.

[0005] (2) Conflict between labeling quality control and automation: To control false label noise, existing systems usually set a fixed high confidence threshold, and only predictions exceeding this threshold are automatically adopted. This results in a large number of predictions in the "gray area," which are slightly uncertain but may be correct, being sent to the manual review queue, resulting in low automation. At the same time, the system lacks the ability to dynamically adapt to labelers of different levels and cannot flexibly adjust the trust strategy according to the overall labeling quality, which can easily lead to noise accumulation or efficiency loss.

[0006] (3) Catastrophic forgetting in incremental learning: When a model is trained (fine-tuned) on a new batch of labeled data, it will severely overwrite and forget the knowledge it has previously learned, resulting in a sharp decline in performance for labeled categories. This phenomenon is called "catastrophic forgetting". This makes it impossible for the model to maintain stable performance in continuous labeling, forcing developers to retrain the model with all historical data periodically, losing the fundamental meaning of incremental learning, and also causing a huge waste of computing resources.

[0007] In summary, existing technologies for active annotation of multimodal data suffer from systemic flaws such as "blind initiation, rigid selection, and severe forgetting," making it difficult to balance annotation costs, data quality, and model stability. Therefore, there is an urgent need for an integrated solution to truly achieve efficient, high-quality, and sustainable intelligent data annotation. Summary of the Invention

[0008] The purpose of this invention is to address the shortcomings of the existing technologies by providing an efficient, high-quality, and sustainable method for proactive annotation of multimodal data based on incremental learning. This method enables adaptive cold start, intelligent quality control, and stable incremental learning.

[0009] The technical solution of this invention is implemented as follows: a multimodal data active annotation method based on incremental learning, the method comprising the following steps:

[0010] (S1) Cold start: Based on the information entropy characteristics of the unlabeled multimodal dataset, the initial labeled sample size is dynamically determined, the initial training set is obtained, and the basic labeled model is trained.

[0011] (S2) Intelligent filtering: Predict pseudo-labels for unlabeled data using the current labeling model, generate dynamic confidence thresholds based on the comprehensive trust level of the system's user group, and then divide the pseudo-labels into an automatically adopted set and a manually reviewed set.

[0012] (S3) Hard sample mining: Receive the correction operation of the pseudo-labels in the manual review set, calculate the prediction-correction difference Δ of modality adaptation, identify hard samples based on Δ and store them in the structured hard sample pool.

[0013] (S4) Anti-forgetting incremental training: Start a new round of model training, integrate the newly added real labels and anchor samples from the hard sample pool into the training set, and consolidate the regularization term by combining knowledge distillation constraints and elastic weights in the loss function to build an anti-forgetting learning mechanism for continuous annotation scenarios;

[0014] (S5) Closed-loop optimization: Monitor the memory strength decay of samples in the difficult sample pool and the manual correction mode, dynamically adjust the composition and training weight of samples in the pool, and feed the optimization signal back to the screening threshold of S2 and the training strategy of S4 to form an adaptive labeling-training closed loop.

[0015] In the above-mentioned multimodal data active annotation method based on incremental learning, the specific steps for dynamically determining the initial labeled sample size in S1 include: calculating the overall information entropy H(D) of the unlabeled dataset D; setting an entropy threshold τ, and when H(D) > τ, adopting an exploratory sampling strategy: N = α·log(|D|);

[0016] When H(D)≤τ, a coverage sampling strategy is adopted: N=β·√|D|;

[0017] Where N is the initial number of labeled samples, |D| is the total amount of the dataset, and α∈[2,5] and β∈[10,30] are configurable coefficients related to the data modality.

[0018] In the above-mentioned multimodal data active annotation method based on incremental learning, the method for generating dynamic confidence thresholds in S2 includes:

[0019] The weighted average trust level of the current user group in the calculation system is L_avg = Σ(w_i·L_i) / Σw_i, where L_i is the historical level of user i, and w_i is the weight related to its historical labeling accuracy.

[0020] Based on the dynamic mapping baseline threshold θ_base of L_avg: when L_avg∈[1,3], θ_base=0.85, when L_avg∈[4,6], θ_base=0.75, and when L_avg∈[7,10], θ_base=0.65; introducing the redundancy control factor ε∈[0.05,0.15], the final dynamic confidence threshold θ=θ_base+ε is obtained.

[0021] In the above-mentioned multimodal data active annotation method based on incremental learning, the specific rule for splitting pseudo-labels as described in S2 is as follows:

[0022] For pseudo-labels with confidence ≥ θ, the main part P is directly converted into a real label and added to the training set, where P ∈ [70%, 90%] and is positively correlated with L_avg;

[0023] For pseudo-labels with confidence ∈ [θ, θ+0.1], even if the threshold requirement is met, they are still forcibly submitted to the manual review queue;

[0024] For pseudo-labels with a confidence level < θ, all are retained in the unlabeled pool.

[0025] In the above-mentioned multimodal data active annotation method based on incremental learning, the calculation method of the modality adaptive prediction-correction difference Δ in S3 includes:

[0026] For image / video object detection tasks: Δ_bbox = 1 - IoU(B_pred, B_revised), where IoU is the intersection-union ratio of the predicted box and the revised box, B_pred is the predicted box, and B_revised is the revised box;

[0027] For text sequence labeling tasks: Δ_text = 1 - (|P_pred∩P_revised| / |P_revised|), where P is the set of labeled entities, P_pred is the set of predicted objects, and P_revised is the set of corrected objects;

[0028] For the 3D point cloud segmentation task: Δ_pointcloud=(1 / N)·Σ_i‖C_pred(i)-C_revised(i)‖, where C is the classification vector or coordinates of the point, C_pred is the predicted classification vector or coordinates of the point, C_revised is the corrected classification vector or coordinates of the point, and ‖C_pred(i)-C_revised(i)‖ represents the absolute distance of the vector or coordinates from the origin.

[0029] In the above-mentioned multimodal data active annotation method based on incremental learning, the difficult sample pool mentioned in S3 is a structured management pool with a capacity limit and a culling mechanism. Its management strategy includes:

[0030] For each sample, maintain a comprehensive weight W = λ_1·Δ + λ_2·(1-Confidence_original) + λ_3·Age_Factor, where λ is the adjustment coefficient, Confidence_original is the original confidence level, and Age_Factor is the historical retention factor;

[0031] When the pool capacity reaches its limit, the sample with the lowest overall weight W is eliminated; for samples already in the pool, if their Δ value in multiple consecutive training rounds remains below the maintenance threshold, their priority is reduced.

[0032] In the above-mentioned multimodal data active annotation method based on incremental learning, the composite anti-forgetting loss function L_total mentioned in S4 is: L_total=L_CE+γ_1·L_KD+γ_2·L_EWC, where L_CE is the standard cross-entropy loss, L_KD is the knowledge distillation loss with the output of the previous model, L_EWC is the elastic weight consolidation regularization term based on Fisher information of parameter importance, and γ_1 and γ_2 are dynamic adjustment coefficients.

[0033] In the above-mentioned multimodal data active annotation method based on incremental learning, the adjustment strategy for the dynamically adjusted coefficients γ_1 and γ_2 includes:

[0034] After each round of training, the retention rate R of the model for historically labeled categories is evaluated on the validation set; if the decrease of R exceeds the preset alarm threshold, the values ​​of γ_1 and γ_2 are increased proportionally to enhance the strength of the anti-forgetting constraint.

[0035] If R remains stable, γ_1 and γ_2 are gradually decreased to avoid over-constraining the learning of new knowledge.

[0036] In the above-mentioned multimodal data active annotation method based on incremental learning, the closed-loop optimization mentioned in S5 includes feedback on the screening threshold, specifically: statistically analyzing the distribution relationship between the original confidence_original and the correction amount Δ_i in the samples corrected from the manual review set in the current round;

[0037] If a large number of samples are found to have an original confidence level of Confidence_original > 0.9 but a correction amount Δ_i > 0.2, an optimization signal is generated, suggesting that the dynamic confidence threshold θ or the redundancy control factor ε be increased in the next round of S2.

[0038] In the above-mentioned multimodal data active annotation method based on incremental learning, the closed-loop optimization in S5 includes feedback on the difficult sample pool, specifically: for each round of training, calculate the Fisher information value F_i of each sample in the pool to measure its importance as a memory anchor.

[0039] Monitor the decay rate of F_i with each training round. If the decay rate of a sample exceeds a set threshold, trigger the reinforcement replay mechanism by increasing its sampling weight, increasing its frequency of occurrence in training rounds, or a combination of both.

[0040] By employing the above method, this invention systematically solves the interconnected pain points in existing active annotation systems, such as blind cold start, contradiction between selection efficiency and quality, and catastrophic model forgetting, by constructing a complete adaptive closed loop that includes "cold start → intelligent screening → difficulty mining → anti-forgetting training → closed-loop optimization". It achieves an upgrade from an open-loop pipeline to an intelligent closed loop, enabling the entire annotation system to self-optimize and become more efficient and stable with use. Attached Figure Description

[0041] The present invention will be further described in detail below with reference to the embodiments shown in the accompanying drawings, but this does not constitute any limitation on the present invention.

[0042] Figure 1 This is a flowchart illustrating the present invention;

[0043] Figure 2 This is a schematic diagram of the simulation results and data analysis of Embodiment 1 of the present invention;

[0044] Figure 3 This is a schematic diagram of the ablation experiment results of the anti-amnesia mechanism in Embodiment 2 of the present invention;

[0045] Figure 4 This is a schematic diagram of clothing stain detection in Embodiment 3 of the present invention. Detailed Implementation

[0046] See Figure 1 As shown, based on a fabric stain recognition case, this invention presents a multimodal data active annotation method based on incremental learning, characterized by the following steps:

[0047] (S1) Cold start: Based on the information entropy characteristics of the unlabeled multimodal dataset, the initial labeled sample size is dynamically determined, the initial training set is obtained, and the basic labeled model is trained.

[0048] The specific steps for dynamically determining the initial labeled sample size include: calculating the overall information entropy H(D) of the unlabeled dataset D; setting an entropy threshold τ, and when H(D) > τ, adopting an exploratory sampling strategy: N = α·log(|D|);

[0049] When H(D)≤τ, a coverage sampling strategy is adopted: N=β·√|D|;

[0050] Where N is the initial number of labeled samples, |D| is the total amount of the dataset, and α∈[2,5] and β∈[10,30] are configurable coefficients related to the data modality.

[0051] By adaptively selecting the initial labeling strategy and sample size (N) based on the overall information entropy (H) of the dataset, this approach overcomes the blindness of relying on experience or random determination of initial samples. Based on data complexity, the scheme intelligently decides whether to prioritize "exploration" (logarithmic sampling for high entropy) or "coverage" (square root sampling for low entropy), thereby laying the most robust model foundation for subsequent learning with the optimal initial labeling cost and improving overall efficiency from the source.

[0052] (S2) Intelligent filtering: Predict pseudo-labels for unlabeled data using the current labeling model, generate dynamic confidence thresholds based on the comprehensive trust level of the system's user group, and then divide the pseudo-labels into an automatically adopted set and a manually reviewed set.

[0053] The method for generating the dynamic confidence threshold includes:

[0054] The weighted average trust level of the current user group in the calculation system is L_avg = Σ(w_i·L_i) / Σw_i, where L_i is the historical level of user i, and w_i is the weight related to its historical labeling accuracy.

[0055] Based on the dynamic mapping baseline threshold θ_base of L_avg: when L_avg∈[1,3], θ_base=0.85, when L_avg∈[4,6], θ_base=0.75, and when L_avg∈[7,10], θ_base=0.65; introducing the redundancy control factor ε∈[0.05,0.15], the final dynamic confidence threshold θ=θ_base+ε is obtained.

[0056] The specific rules for splitting pseudo-tag traffic are as follows:

[0057] For pseudo-labels with confidence ≥ θ, the main part P is directly converted into a real label and added to the training set, where P ∈ [70%, 90%] and is positively correlated with L_avg;

[0058] For pseudo-labels with confidence ∈ [θ, θ+0.1], even if the threshold requirement is met, they are still forcibly submitted to the manual review queue;

[0059] For pseudo-labels with a confidence level < θ, all are retained in the unlabeled pool.

[0060] (S3) Hard sample mining: Receive the correction operation of pseudo-labels in the manual review set, calculate the modality adaptive prediction-correction difference Δ, identify hard samples based on Δ and store them in the structured hard sample pool.

[0061] The method for calculating the modality-adaptive prediction-correction difference Δ includes:

[0062] For image / video object detection tasks: Δ_bbox = 1 - IoU(B_pred, B_revised), where IoU is the intersection-union ratio of the predicted box and the revised box, B_pred is the predicted box, and B_revised is the revised box;

[0063] For text sequence labeling tasks: Δ_text = 1 - (|P_pred∩P_revised| / |P_revised|), where P is the set of labeled entities, P_pred is the set of predicted objects, and P_revised is the set of corrected objects;

[0064] For the 3D point cloud segmentation task: Δ_pointcloud=(1 / N)·Σ_i‖C_pred(i)-C_revised(i)‖, where C is the classification vector or coordinates of the point, C_pred is the predicted classification vector or coordinates of the point, C_revised is the corrected classification vector or coordinates of the point, and ‖C_pred(i)-C_revised(i)‖ represents the absolute distance of the vector or coordinates from the origin.

[0065] The aforementioned difficult sample pool is a structured management pool with a capacity limit and a culling mechanism. Its management strategy includes:

[0066] For each sample, maintain a comprehensive weight W = λ_1·Δ + λ_2·(1-Confidence_original) + λ_3·Age_Factor, where λ is the adjustment coefficient, Confidence_original is the original confidence level, and Age_Factor is the historical retention factor;

[0067] When the pool capacity reaches its limit, the sample with the lowest overall weight W is eliminated; for samples already in the pool, if their Δ value in multiple consecutive training rounds remains below the maintenance threshold, their priority is reduced.

[0068] By designing a structured management strategy for the difficult sample pool, which includes a capacity limit, a comprehensive weighted score (W), and a eviction mechanism, dynamic lifecycle management of difficult samples is achieved. This mechanism retains the most valuable memory anchors and evicts fully learned or invalidated samples, ensuring the long-term effectiveness and storage efficiency of the difficult sample pool and avoiding unnecessary expansion of the pool.

[0069] (S4) Anti-forgetting incremental training: Start a new round of model training, integrate the newly added real labels and anchor samples from the hard sample pool into the training set, and consolidate the regularization term by combining knowledge distillation constraints and elastic weights in the loss function to build an anti-forgetting learning mechanism for continuous annotation scenarios.

[0070] The composite anti-forgetting loss function L_total is: L_total=L_CE+γ_1·L_KD+γ_2·L_EWC, where L_CE is the standard cross-entropy loss, L_KD is the knowledge distillation loss with the output of the previous model, L_EWC is the elastic weight consolidation regularization term based on Fisher information of parameter importance, and γ_1 and γ_2 are dynamic adjustment coefficients.

[0071] The adjustment strategies for the dynamic adjustment coefficients γ_1 and γ_2 include:

[0072] After each round of training, the retention rate R of the model for historically labeled categories is evaluated on the validation set. If the decrease of R exceeds a preset alarm threshold, the values ​​of γ_1 and γ_2 are increased proportionally to enhance the strength of the anti-forgetting constraint. The alarm threshold is preferably 5%-10%. Here, the value of R can be set to any number from 0 to 1 according to actual needs. If the value of R is 1, the values ​​of γ_1 and γ_2 are continuously increased to enhance the strength of the anti-forgetting constraint. The lower the value of R, the faster the values ​​of γ_1 and γ_2 decrease, until the latest learned knowledge becomes the main focus.

[0073] If R remains stable, γ_1 and γ_2 are gradually decreased to avoid over-constraining the learning of new knowledge.

[0074] (S5) Closed-loop optimization: Monitor the memory strength decay of samples in the difficult sample pool and the manual correction mode, dynamically adjust the composition and training weight of samples in the pool, and feed the optimization signal back to the screening threshold of S2 and the training strategy of S4 to form an adaptive labeling-training closed loop.

[0075] The closed-loop optimization includes feedback on the screening threshold, specifically: statistically analyzing the distribution relationship between the original confidence level Confi and the correction amount Δ_i in the samples corrected from the manual review set in the current round;

[0076] If a large number of samples are found to have an original confidence level Conf > 0.9 but a correction amount Δ_i > 0.2, an optimization signal is generated, suggesting that the dynamic confidence threshold θ or the redundancy control factor ε be increased in the next round S2.

[0077] The closed-loop optimization includes feedback on the difficult sample pool, specifically: for each round of training, calculate the Fisher information value F_i of each sample in the pool to measure its importance as a memory anchor.

[0078] Monitor the decay rate of F_i with each training round. If the decay rate of a sample exceeds a set threshold, trigger the reinforcement replay mechanism by increasing its sampling weight, increasing its frequency of occurrence in training rounds, or a combination of both. The preferred threshold is 10%-50%.

[0079] By innovatively combining knowledge distillation constraint (L_KD) and elastic weight consolidation regularization term (L_EWC) in the loss function, a two-level, three-dimensional anti-forgetting protection is achieved, from the model's "output distribution stability" to "parameter importance consolidation".

[0080] Furthermore, by introducing a dynamic adjustment coefficient strategy based on the historical task performance retention rate (R), the system achieves adaptive optimization of the anti-forgetting strength: when performance degradation is detected, the system can automatically strengthen constraints to consolidate memory, and weaken constraints when the state is stable to avoid hindering the learning of new knowledge, thus intelligently balancing the contradiction between "not forgetting the old" and "learning the new quickly".

[0081] Meanwhile, by establishing a closed-loop feedback, the front-end quality monitoring signal (high-confidence, high-correction samples) and the back-end memory state monitoring signal (Fisher information decay of difficult samples) are fed back to the screening threshold and sample replay strategy, respectively, realizing the deep coupling and collaborative evolution of the anti-forgetting core engine and the two peripheral systems of annotation quality control and sample resource management.

[0082] Ultimately, through the synergistic operation of the aforementioned technologies, an intelligent closed-loop system with self-sensing, diagnostic, and optimization capabilities was achieved.

[0083] Example 1: Overall Verification Based on Simulation Experiments

[0084] To systematically verify the comprehensive technical effects of the method described in this invention, a verification simulation experiment was designed and executed. The core purpose of this simulation experiment is not to accurately predict absolute performance values, but to verify the relative advantages of this invention compared to existing technical methods within a fair and consistent comparative framework.

[0085] 1. Description of Experimental Nature: This experiment is a computer simulation experiment. By establishing a mathematical model that conforms to well-known principles in the field of machine learning, it simulates the entire incremental labeling process under different strategies. The simulation parameters are set based on common knowledge in the field, aiming to control variables and focus on verifying the inherent logical validity of the technical solution described in this invention.

[0086] 2. Experimental Design and Comparison Baseline

[0087] Simulated task: Simulate an object detection task with 10 categories, divided into 5 consecutive incremental learning stages to simulate a continuous annotation scenario.

[0088] Comparison Methods: Two representative baseline methods were set as comparison benchmarks: (1) Random sampling + fine-tuning: representing the most basic annotation and learning strategy. (2) Uncertainty-based active learning + fine-tuning: representing the commonly used techniques that focus on sample efficiency. (3) The method of this invention: fully implementing the process described in this invention.

[0089] Evaluation metrics: cumulative amount of manual annotation (measures efficiency), average precision of the final model (mAP, measures performance), and average retention rate of historical classes (measures resistance to forgetting).

[0090] 3. Simulation results and data analysis, please refer to the following for details. Figure 2 Multiple independent simulations all showed a stable trend, with one typical result shown in the table below:

[0091]

[0092] 4. Experimental conclusions and technical effect verification The simulation data clearly verified the technical effect of the present invention: (1) Significantly improved annotation efficiency: Under the premise of achieving the same target performance (mAP=0.95), the cumulative annotation amount required by the present invention is only 49.8% of the active learning baseline and 35.4% of the random sampling baseline, which proves the effectiveness of the adaptive sampling and intelligent screening mechanism. (2) Excellent model stability: The present invention controls the average forgetting rate of the model for the learned knowledge to 4.2%, which is much lower than the active learning baseline (17.7%) and the random sampling baseline (25.1%), which verifies the synergistic effect of difficult sample mining and composite anti-forgetting training mechanism. (3) Clear comprehensive advantages: The simultaneous and significant improvement in efficiency and stability proves that the closed-loop system is not a simple superposition of multiple modules, but produces a synergistic gain of "1+1>2", which solves the inherent contradiction of "improving efficiency at the expense of stability" in the prior art.

[0093] 5. The effectiveness of the simulation experiment demonstrates that this simulation experiment, by controlling a single variable (method strategy), fairly compared the inherent potential of different technical approaches. Although the experimental environment was simulated, the significant and consistent relative advantage trend revealed is sufficient to prove the progressiveness and practicality of the technical solution of this invention. This trend conforms to the basic principles of machine learning (such as high-quality samples leading to higher learning efficiency, and regularization constraints helping to stabilize the model), and is understood and recognized by those skilled in the art.

[0094] Example 2:

[0095] Anti-forgetting verification based on simulated ablation experiments

[0096] To verify the effectiveness of the combined knowledge distillation constraint (L_KD) and elastic weight consolidation regularization term (L_EWC) in the loss function in providing a two-tiered, three-dimensional protection against forgetting—from output distribution stability to parameter importance consolidation—multi-round training comparison experiments were conducted on the COCO 2017 dataset. The experiments aimed to evaluate the impact of different strategies on model stability during continuous training.

[0097] 1. Experimental Setup

[0098] Dataset: A subset of the COCO 2017 dataset was used, selecting 10 common object detection categories: people, bicycles, cars, motorcycles, airplanes, buses, trains, trucks, boats, and birds. All categories are visible in each training round.

[0099] Model and training configuration:

[0100] Base model: YOLOv5s, ImageNet pre-trained initialization

[0101] Optimizer: SGD, initial learning rate 0.001

[0102] Training rounds: A total of 5 training rounds, each consisting of 10 epochs.

[0103] Performance is evaluated on a fixed test set after each round of training.

[0104] 2. Experimental results are as follows Figure 3 As shown.

[0105] The global mAP evolution of each strategy after 5 rounds of training is as follows:

[0106]

[0107] Comparison of average forgetting rates:

[0108]

[0109] 3. Experimental Conclusions

[0110] Experimental results show that the proposed solution can effectively maintain model performance during continuous training, controlling the average forgetting rate at 5.46%, which is significantly better than other comparative strategies, verifying the effectiveness of the above-mentioned combination of technologies.

[0111] Example 3:

[0112] Fabric stain detection model generation based on the method of this invention (single-person annotation scenario)

[0113] This embodiment uses fabric stain detection in the textile industry as an application scenario to explain in detail how to efficiently generate a high-precision stain detection model using the method described in this invention under single-person annotation conditions.

[0114] 1. Scenario Setting and Data Preparation

[0115] Dataset: Contains 29,250 fabric images with a resolution of 640×640, categorized as stains as defects. All images are unlabeled, forming the initial unlabeled pool D, with a total of |D|=29,250. Preset entropy threshold τ=0.5, β=18, redundancy control factor ε=0.08, hard sample threshold Δ_th=0.3, and 500 labeled images reserved as a validation set.

[0116] Target annotation: Detect and annotate the stain areas in each image, and output the bounding boxes of the stains.

[0117] Labeling personnel: 1 experienced quality inspector with a historical labeling accuracy rate of 96%, whose user level is set to L=10 based on the user level mapping (L∈[7,10]).

[0118] 2. Specific Implementation Steps

[0119] S1: Cold Start – Adaptive Determination of Initial Labeled Sample Size

[0120] Calculate the overall information entropy H(D) of the unlabeled dataset D. Based on the entropy values ​​of the image color histogram and texture features, H(D) = 0.48 (lower than the preset entropy threshold τ = 0.5).

[0121] Since H(D)≤τ, a coverage sampling strategy is adopted: N = β·√|D, then N=18·√29250≈18×171.0≈3,078, which is rounded to 3,000 as the initial labeled samples.

[0122] 3,000 images were randomly selected from the dataset and manually labeled by annotators to obtain the initial training set. These samples were used to train the YOLO11n model, resulting in the base annotation model M0, with an initial detection mAP of 0.42.

[0123] Beneficial effects: Using a traditional fixed initial annotation amount requires a huge investment of resources. This invention determines data complexity based on information entropy, and obtains a working initial model with only 3,000 images (about 10% of the data), significantly reducing the cost of cold start.

[0124] S2: Intelligent Filtering – Dynamic Threshold Triage Pseudo-Labels

[0125] Using the current model M0, we predict pseudo-labels for the remaining 26,250 images in the unlabeled pool, generating bounding boxes and confidence scores for each image.

[0126] User trust level: Single person annotation, L_avg=10 (based on historical accuracy of 96%, corresponding to level 10).

[0127] Calculate the dynamic threshold θ: When L_avg∈[7,10], θ_base=0.65, so θ=θ_base+ε=0.65+0.08=0.73.

[0128] 5000 images were randomly selected from the unlabeled dataset, and pseudo-labels were generated using an annotation model and then distributed among the datasets.

[0129] Samples with a confidence level ≥ 0.73: 1,832 images in total. 1,594 images were automatically converted to true labels based on P = 0.82 + L_avg / 20 = 0.82 + 10 / 20 = 0.87 and directly added to the training set; the remaining 238 images with a confidence level in the range [0.73, 0.83) were subject to mandatory manual review.

[0130] Samples with confidence level <0.73: 3,168 images were all retained in the unlabeled pool.

[0131] Beneficial effects: The dynamic threshold automatically lowers the threshold based on the expert's level, enabling more high-confidence pseudo-labels to be automatically adopted. The amount of manual review accounts for only 4.8% (238 / 5,000) of pseudo-labels, greatly improving efficiency.

[0132] S3: Difficult Sample Discovery – Identifying and Managing Difficult Samples

[0133] 238 forced submissions of fake labels were manually reviewed to obtain corrected, genuine labels.

[0134] Calculate the correction value Δ for each verification image (object detection task): For each predicted bounding box in each image, Δ = 1 - IoU(predicted bounding box, corrected bounding box), and take the average Δ of all predicted bounding boxes in the image as the Δ value of that image. Images with Δ > Δ_th are marked as hard samples and stored in the hard sample pool (56 images in this batch).

[0135] Difficult sample pool management:

[0136] The capacity limit is set to 500. Each sample maintains a comprehensive weight W = λ1·Δ + λ2·(1-Confidence_original) + λ3·Age_Factor, with λ1=0.5, λ2=0.3, and λ3=0.2; the original confidence level is the predicted value; Age_Factor is initially 1 and increases by 0.1 each round.

[0137] When the pool is full, the sample with the lowest W is eliminated; if a sample has Δ < 0.1 for 3 consecutive rounds, its priority is reduced.

[0138] Beneficial effects: The difficult sample pool dynamically retains the most valuable difficult examples, and 56 high-Δ samples are stored first. They can be replayed in subsequent training to enhance the model's ability to recognize complex stain patterns.

[0139] S4: Anti-forgetting incremental training – continuous learning of new annotations

[0140] The current training set is formed by integrating the newly added real labels (1,594 automatically adopted labels + 238 manually reviewed labels) with the anchor samples (currently 56 labels) in the hard sample pool.

[0141] Incremental training: Starting with model M0, use the composite loss function L_total = L_CE + γ1·L_KD+ γ2·L_EWC.

[0142] Initially, γ1 = 0.3 and γ2 = 0.5.

[0143] Dynamically adjust γ1 and γ2: After each training round, evaluate the model's retention rate R for previously learned stain detection performance on the validation set (500 reserved labeled images). In this example, after the first round of incremental adjustment, R decreases by 3% (below 5%), and γ1 and γ2 decrease by 5% to 0.285 and 0.475, respectively; after the second round, R decreases by 6%, triggering enhancement, and γ1 and γ2 increase by 20% to 0.342 and 0.57, respectively.

[0144] Beneficial effects: Dynamic adjustment enables the model to adaptively balance the consolidation of learned stain features with the learning of new stain morphologies, avoiding the forgetting of existing stain detection capabilities due to the introduction of new data.

[0145] S5: Closed-loop optimization – feedback tuning of system parameters

[0146] Threshold Feedback: In this round of manual review, samples with an initial confidence level (Conf) > 0.9 but a correction value (Δ) > 0.2 were analyzed. Eight such samples were found (3.4% of the reviewed samples), indicating that the current threshold θ = 0.73 might be slightly lenient. An optimization signal was generated, and in the next round, θ will be increased by 0.02 to 0.75, and the redundancy control factor ε will be adjusted from 0.08 to 0.09.

[0147] Feedback from the difficult sample pool: Monitor the decay rate of Fisher information value F_i of samples in the pool. It was found that the decay rate of a sample dropped from 0.48 to 0.36 in two consecutive rounds, a decrease of more than 20%, triggering reinforcement replay: in the next round of training, the sampling weight of this sample is increased by 50%, and its frequency of occurrence increases.

[0148] The next round of training adopted the above feedback. After the threshold was raised, the automatic adoption rate dropped to 85%, but the accuracy of the automatically adopted samples increased from 93% to 97%, and the phenomenon of high confidence and high correction was reduced to 2%, further optimizing human-machine collaboration.

[0149] 3. Experimental Results and Comparison

[0150] After 5 rounds of incremental training (with approximately 6,000 new annotations added), the final model's performance on the test set (including stain categories, 2,000 images) is as follows:

[0151]

[0152] Annotation efficiency: This invention saves 50% of the manual annotation work compared to traditional methods (6,000 vs 12,000).

[0153] Model performance: The final mAP reached 0.86, which was significantly higher than the comparison method.

[0154] Anti-forgetting: The average performance decay rate is only 3.8%, while the traditional method is as high as 21.5%, proving that the present invention effectively suppresses the forgetting of learned stain features during incremental learning.

[0155] Closed-loop optimization: Through threshold feedback, the accuracy of automatic adoption is improved from 93% to 97%, reducing noise accumulation; replay of difficult samples increases the contribution of key samples by 30%.

[0156] 4. Conclusion

[0157] This embodiment fully implements the method of the present invention in a fabric stain detection task with single-person annotation and a dataset size of 29,250 images. Through adaptive cold start (S1), dynamic threshold screening (S2), hard sample mining (S3), anti-forgetting training (S4), and closed-loop optimization (S5), the manual annotation cost is significantly reduced (saving more than 50%), while improving model accuracy (mAP 0.86) and learning stability (performance decay rate <4%), verifying the practicality and inventiveness of the present invention in industrial quality inspection scenarios.

[0158] The above-described embodiments are preferred embodiments of the present invention and are only used to facilitate the illustration of the present invention. They are not intended to limit the present invention in any way. Any person skilled in the art who makes local modifications or alterations to the technical content disclosed in the present invention without departing from the scope of the technical features of the present invention shall still fall within the scope of the technical features of the present invention.

Claims

1. A method for active annotation of multimodal data based on incremental learning, characterized in that, The method includes the following steps: (S1) Cold start: Based on the information entropy characteristics of the unlabeled multimodal dataset, the initial labeled sample size is dynamically determined, the initial training set is obtained, and the basic labeled model is trained. (S2) Intelligent filtering: Predict pseudo-labels for unlabeled data using the current labeling model, generate dynamic confidence thresholds based on the comprehensive trust level of the system's user group, and then divide the pseudo-labels into an automatically adopted set and a manually reviewed set. (S3) Hard sample mining: Receive the correction operation of the pseudo-labels in the manual review set, calculate the prediction-correction difference Δ of modality adaptation, identify hard samples based on Δ and store them in the structured hard sample pool. (S4) Anti-forgetting incremental training: Start a new round of model training, integrate the newly added real labels and anchor samples from the hard sample pool into the training set, and consolidate the regularization term by combining knowledge distillation constraints and elastic weights in the loss function to build an anti-forgetting learning mechanism for continuous annotation scenarios; (S5) Closed-loop optimization: Monitor the memory strength decay of samples in the difficult sample pool and the manual correction mode, dynamically adjust the composition and training weight of samples in the pool, and feed the optimization signal back to the screening threshold of S2 and the training strategy of S4 to form an adaptive labeling-training closed loop.

2. The multimodal data active annotation method based on incremental learning according to claim 1, characterized in that, The specific steps for dynamically determining the initial labeled sample size as described in S1 include: calculating the overall information entropy H(D) of the unlabeled dataset D; setting an entropy threshold τ, and when H(D) > τ, adopting an exploratory sampling strategy: N = α·log(|D|); When H(D)≤τ, a coverage sampling strategy is adopted: N=β·√|D|; Where N is the initial number of labeled samples, |D| is the total amount of the dataset, and α∈[2,5] and β∈[10,30] are configurable coefficients related to the data modality.

3. The multimodal data active annotation method based on incremental learning according to claim 1, characterized in that, The method for generating dynamic confidence thresholds as described in S2 includes: The weighted average trust level of the current user group in the calculation system is L_avg = Σ(w_i·L_i) / Σw_i, where L_i is the historical level of user i, and w_i is the weight related to its historical labeling accuracy. Based on the dynamic mapping baseline threshold θ_base of L_avg: when L_avg∈[1,3], θ_base=0.85, when L_avg∈[4,6], θ_base=0.75, and when L_avg∈[7,10], θ_base=0.65; introducing the redundancy control factor ε∈[0.05,0.15], the final dynamic confidence threshold θ=θ_base+ε is obtained.

4. A method for active annotation of multimodal data based on incremental learning according to claim 1 or 3, characterized in that, The specific rules for splitting pseudo-tag traffic as described in S2 are as follows: For pseudo-labels with confidence ≥ θ, the main part P is directly converted into a real label and added to the training set, where P ∈ [70%, 90%] and is positively correlated with L_avg; For pseudo-labels with confidence ∈ [θ, θ+0.1], even if the threshold requirement is met, they are still forcibly submitted to the manual review queue; For pseudo-labels with a confidence level < θ, all are retained in the unlabeled pool.

5. The multimodal data active annotation method based on incremental learning according to claim 1, characterized in that, The method for calculating the modal adaptive prediction-correction difference Δ described in S3 includes: For image / video object detection tasks: Δ_bbox = 1 - IoU(B_pred, B_revised), where IoU is the intersection-union ratio of the predicted box and the revised box, B_pred is the predicted box, and B_revised is the revised box; For text sequence labeling tasks: Δ_text = 1 - (|P_pred∩P_revised| / |P_revised|), where P is the set of labeled entities, P_pred is the set of predicted objects, and P_revised is the set of corrected objects; For the 3D point cloud segmentation task: Δ_pointcloud=(1 / N)·Σ_i‖C_pred(i)-C_revised(i)‖, where C is the classification vector or coordinates of the point, C_pred is the predicted classification vector or coordinates of the point, C_revised is the corrected classification vector or coordinates of the point, and ‖C_pred(i)-C_revised(i)‖ represents the absolute distance of the vector or coordinates from the origin.

6. The multimodal data active annotation method based on incremental learning according to claim 1, characterized in that, The difficult sample pool described in S3 is a structured management pool with a capacity limit and a elimination mechanism. Its management strategies include: For each sample, maintain a comprehensive weight W = λ_1·Δ + λ_2·(1-Confidence_original) + λ_3·Age_Factor, where λ is the adjustment coefficient, Confidence_original is the original confidence level, and Age_Factor is the historical retention factor; When the pool capacity reaches its limit, the sample with the lowest overall weight W is eliminated; for samples already in the pool, if their Δ value in multiple consecutive training rounds remains below the maintenance threshold, their priority is reduced.

7. The multimodal data active annotation method based on incremental learning according to claim 1, characterized in that, The composite anti-forgetting loss function L_total described in S4 is: L_total=L_CE+γ_1·L_KD+γ_2·L_EWC, where L_CE is the standard cross-entropy loss, L_KD is the knowledge distillation loss with the output of the previous model, L_EWC is the elastic weight consolidation regularization term based on Fisher information of parameter importance, and γ_1 and γ_2 are dynamic adjustment coefficients.

8. The multimodal data active annotation method based on incremental learning according to claim 7, characterized in that, The adjustment strategies for the dynamic adjustment coefficients γ_1 and γ_2 include: After each round of training, the retention rate R of the model for historically labeled categories is evaluated on the validation set; if the decrease of R exceeds the preset alarm threshold, the values ​​of γ_1 and γ_2 are increased proportionally to enhance the strength of the anti-forgetting constraint. If R remains stable, γ_1 and γ_2 are gradually decreased to avoid over-constraining the learning of new knowledge.

9. The multimodal data active annotation method based on incremental learning according to claim 1, characterized in that, The closed-loop optimization described in S5 includes feedback on the screening threshold, specifically: statistically analyzing the distribution relationship between the original confidence level Confi and the correction amount Δ_i in the samples corrected from the manual review set in the current round; If a large number of samples are found to have an original confidence level Conf > 0.9 but a correction amount Δ_i > 0.2, an optimization signal is generated, suggesting that the dynamic confidence threshold θ or the redundancy control factor ε be increased in the next round S2.

10. The multimodal data active annotation method based on incremental learning according to claim 1, characterized in that, The closed-loop optimization described in S5 includes feedback to the hard sample pool, specifically: for each round of training, calculate the Fisher information value F_i of each sample in the pool to measure its importance as a memory anchor. Monitor the decay rate of F_i with each training round. If the decay rate of a sample exceeds a set threshold, trigger the reinforcement replay mechanism by increasing its sampling weight, increasing its frequency of occurrence in training rounds, or a combination of both.

Images