Incremental labeling method based on large and small model collaboration, electronic equipment and medium
By combining large and small models and using replayed images and disambiguated labels to generate candidate labels, the high cost and complexity of the data labeling process are solved, efficient and accurate data labeling is achieved, and the level of automation is improved.
Patent Information
- Application Number
- CN202510727711.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-03
- Publication Date
- 2025-09-12
AI Technical Summary
Existing technologies have high costs and complexity in the data labeling process, especially in application scenarios that rely on professional knowledge. It is difficult to effectively use large models for automated data labeling. Large models have insufficient performance in specific field tasks, and small models have limited flexibility.
By combining large models and small models, using the large model for preliminary labeling, using replayed images and disambiguation labels to generate candidate labels, generating accurate labels through classifier training, and combining classification loss, knowledge distillation loss and consistency regularization loss for joint training, incremental labeling is achieved.
It achieves efficient and accurate data labeling in the absence of labeled data, reduces labeling costs, improves the level of automation, and combines the versatility of large models with the domain adaptability of small models.
Smart Images
Figure CN120635900A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of machine learning, and in particular to an incremental annotation method, electronic device, and medium based on collaboration between large and small models. Background Art
[0002] With the continuous emergence of data, massive data resources have provided unprecedented rich materials for various research and applications. However, in the process of data development and utilization, manual labeling is often an indispensable link. This process not only consumes a lot of time and economic costs, but also, especially in some practical application scenarios that are highly dependent on professional knowledge (such as medical diagnosis, industrial testing, etc.), its complexity and professionalism further increase the difficulty of labeling. To meet this challenge, researchers have proposed a variety of methods to reduce the burden of labeling, such as semi-supervised learning and noisy label learning. These methods have alleviated the dependence on large-scale labeled data to a certain extent, but still require a certain degree of manual participation or pre-designed labeling strategies to start the model learning process. Therefore, in the context of continuous data growth, how to further reduce the cost of labeling and improve the level of automation remains an important topic in current research.
[0003] The emergent power of large models has sparked widespread interest in prompt-based zero-shot learning in both academia and industry. Unlike traditional methods that fine-tune on large amounts of downstream task data, zero-shot learning can demonstrate strong performance even in the absence of labeled task data. This property has made the effective use of large models for automated data labeling a hot research topic.
[0004] While large models excel in general capabilities, their performance in specific domain tasks often underperforms that of specially optimized domain expert models. Furthermore, due to their closed-source nature and high computational resource requirements, some large models struggle to adapt to specific tasks through continuous training, limiting their potential for further evolution into domain experts. In contrast, small models, with their lower computational costs and greater flexibility, can be gradually optimized through training to adapt to specific task requirements. Therefore, combining the versatility of large models with the evolutionary nature of small models to achieve more efficient and accurate automated data labeling has become a critical scientific issue that needs to be addressed. Summary of the Invention
[0005] In view of the deficiencies of the prior art, the present invention provides an incremental annotation method, electronic device, and medium based on collaboration between large and small models.
[0006] In a first aspect, an embodiment of the present invention provides an incremental annotation method based on large and small model collaboration, the method comprising the following steps:
[0007] Input the unlabeled image to be classified into the large model to predict a set of candidate labels; input the unlabeled image data to be classified and the set of candidate labels into the pre-trained classifier to obtain the label results;
[0008] The training process of the classifier includes:
[0009] For the tth round of training, in response to the unlabeled images of the current training round, the large model predicts a set of candidate labels as the initial candidate label set;
[0010] Calculate the prototype based on the replayed images, classify the unlabeled images of the current training round according to the prototype, and add the classification results to the candidate label set;
[0011] Calculate the temporary class prototype based on the candidate label set, calculate the similarity distribution between the unlabeled images of the current training round and the temporary class prototype to obtain the first label; calculate the second label based on the candidate label set; and use the weighted sum of the first label and the second label as the disambiguation label;
[0012] The classification loss, knowledge distillation loss, and consistency regularization loss are calculated based on the disambiguation labels, unlabeled images of the current training round, and replayed images. The sum of the classification loss, knowledge distillation loss, and consistency regularization loss is used as the joint loss function, and the classifier is trained based on the joint loss function.
[0013] In a second aspect, an embodiment of the present invention provides an electronic device, including:
[0014] at least one processor; and
[0015] a memory communicatively connected to the at least one processor; wherein,
[0016] The memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to execute the above-mentioned incremental annotation method based on large and small model collaboration.
[0017] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned incremental annotation method based on collaboration between large and small models when executed by a processor.
[0018] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which implements the above-mentioned incremental annotation method based on collaboration between large and small models when executed by a processor.
[0019] Compared with the prior art, the present invention has the following beneficial effects:
[0020] The present invention provides an incremental annotation method based on the collaboration of large and small models. By utilizing a large model for automated data annotation, the versatility of the large model and the domain adaptability of the small model are combined, and applied to the field of incremental annotation, thereby alleviating the cost dilemma of incremental annotation. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0022] Figure 1 A schematic diagram of a flow chart of an incremental annotation method based on collaboration between large and small models provided by an embodiment of the present invention;
[0023] Figure 2 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0024] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0025] It should be noted that, unless there is any conflict, the features in the following embodiments and implementations may be combined with each other.
[0026] like Figure 1 As shown, an embodiment of the present invention provides an incremental annotation method based on collaboration between large and small models, the method comprising the following steps:
[0027] The unlabeled image data to be classified is first input into the large model to predict and output a set of candidate labels; the unlabeled image data to be classified and the set of candidate labels output by the large model are input into the pre-trained classifier to obtain the label results of the unlabeled image data to be classified;
[0028] Furthermore, the unlabeled image data x to be classified is input into the large model to obtain a set of candidate labels S, and then the unlabeled image data x to be classified and the set of candidate labels S are input into the pre-trained classifier f(·) for prediction, and the label result is output. The expression is as follows:
[0029]
[0030] Furthermore, the classifier training process includes:
[0031] Step S1: For the new unlabeled pictures that appear in the current stage, the large model predicts a set of possible candidate labels as the candidate label set S of the picture. i .
[0032] Specifically, for data stream T is the total number of training rounds, t is the tth round of training, where each subset Contains N t samples, x i represents the i-th sample. i ∈X represents a sample in the input space. Sample x i The corresponding true label y i ∈Y t Unable to obtain is unknown.
[0033] For the label space, the data distribution is as follows: (i) most samples with new labels gradually appear, Y t represents the label space of the current t-th round of training, represents the newly added label space in the current t-th round of training relative to the t-1th round of training; (ii) Y t The data samples of each category in the subset D t Appeared in.
[0034] At stage t, the goal is to t We train a model f in , without forgetting previous data, where f consists of a feature extraction backbone φ and additional fully connected layers.
[0035] During training, since we cannot access the true labels, we need a large model to calculate the i Predict a set of candidate labels S i , that is, using the large model to label samples, in this example, the five most likely labels for the samples are given as follows:
[0036] S i =M(x i )
[0037] in, M(.) represents the large model. In this example, the large model GPT-4o is used for prediction.
[0038] Step S2: Calculate the prototype based on the replayed image, classify the unlabeled images of the current training round according to the prototype, and add the classification results to the candidate label set S i middle.
[0039] It should be noted that the replay data has a disambiguation label. In this example, the catastrophic forgetting problem of the model in continuous learning is solved by combining replay data with disambiguation labels.
[0040] Specifically, the replayed image is classified with the help of disambiguation labels to generate the category c∈Y t The corresponding prototype μ c , that is, calculate the feature mean of samples of the same category, the expression is as follows:
[0041]
[0042] in, represents the sample feature set belonging to category c; φ(.) represents the feature extractor used to output sample x i The feature vector of , j represents the jth candidate label predicted by the large model, S i Indicates that the large model responds to sample x i A set of candidate labels for prediction.
[0043] In the subsequent stages, the momentum of the prototype is updated, and the expression is as follows:
[0044]
[0045] Among them, γ>0 is a hyperparameter.
[0046] The prototype condenses the knowledge of the old class, so it can relatively accurately identify the old class category that the image may correspond to. Therefore, the prototype is used to predict the possible old class label of the image to supplement the candidate label set to avoid the large model missing the correct sample label. The specific calculation is as follows:
[0047]
[0048] S i =S i ∪{c *}
[0049] Among them, μ j represents the prototype corresponding to category j, c * It is the category corresponding to the sample and the prototype that is closest to it.
[0050] Step S3: Calculate a temporary class prototype based on the candidate label set, calculate the similarity distribution between the unlabeled images of the current training round and the temporary class prototype, and obtain a first label.
[0051] Based on the candidate tag set S updated in step S2 i Compute a temporary class prototype. Specifically, aggregate all samples containing the class label:
[0052]
[0053] Among them, P c ={φ(x i )|c∈S i We hope that the label can align with the temporary class prototype, so we use the softmax function to calculate the similarity distribution between the sample and the temporary class prototype to get the first label
[0054]
[0055] Where j∈S i ,f(x i ) indicates that the classifier is for sample x i Output label result, μ′ j represents the temporary prototype corresponding to category j, μ′ k Represents the temporary class prototype corresponding to category k.
[0056] Step S4: Obtain a second label based on the candidate label set; and use the weighted sum of the first label and the second label as the disambiguation label.
[0057] Initialize pseudo labels based on the new candidate label set
[0058]
[0059] Where, Represents the classifier for sample x i The prediction result on the jth category, I(j∈S i ) indicates whether the jth label exists in the updated candidate label set S i .
[0060] In subsequent rounds, the second label is based on the momentum mechanism.
[0061]
[0062] Where z i is a one-hot vector, i.e. f j(x i ) represents the sample x i The confidence score of label j after classifier processing, β represents the weight.
[0063] Calculate the weighted sum of the first label and the second label to obtain the disambiguation label; the expression is:
[0064]
[0065] In the formula, δ represents the weight.
[0066] In step S5, the classification loss, knowledge distillation loss, and consistency regularization loss are calculated based on the disambiguation labels, the unlabeled images of the current training round, and the replayed images. The sum of the classification loss, knowledge distillation loss, and consistency regularization loss is used as the joint loss function, and the classifier is trained based on the joint loss function.
[0067] Based on the disambiguation label p i Calculate the classification loss:
[0068]
[0069] Where p ij Represents sample x i The label on the jth category;
[0070] The classifier obtained by training t-1 rounds Distill the knowledge to the current classification model f and calculate the knowledge distillation loss:
[0071]
[0072] Where, Represents sample x i The confidence score of the classifier processed on label j after t-1 rounds of training;
[0073] Regarding consistency regularization, given an image x i , generating a weakly enhanced view and a strongly enhanced view During the training phase, it is necessary to ensure strong enhancement of the view The prediction result is as close as possible to the weak enhancement view The predicted probability is expressed as follows:
[0074]
[0075] Where, Represents a weakly enhanced view The prediction result on the jth category, Represents a strongly enhanced view The prediction result on the j-th category.
[0076] Step S6: After the training of the current stage is completed, the representative pictures that are closer to the prototype are selected as replay pictures to participate in the training in the subsequent stage;
[0077] In round t, use M t Represents the entire memory bank, whose maximum storage capacity is m = 2000. First, connect the training set with the previous memory bank to obtain D t ′ =D t ∪M t-1 Then perform sample selection based on intra-class distance and collect samples with the shortest distance to the corresponding prototype:
[0078]
[0079] in Divide the entire set according to the predicted category and then select the N closest to the prototype r This provides richer category information for subsequent learning and alleviates the forgetting of old categories.
[0080] Example 1
[0081] In this example, assume an unlabeled image appears at stage t. Assume the true label for this unlabeled image is "airplane," and the label set at the current stage is {airplane, bridge, seagull, aircraft carrier, fishing boat, castle, whale}. The pixel values of this test example are used as the input image x. This is fed into the large model, which outputs a set of candidate labels.
[0082] Assume that the candidate labels are {plane, bridge, seagull, fishing boat, whale}, that is, S = {1, 2, 3, 5, 7}. Input the image x and the candidate labels S into the pre-trained classifier and predict the output label result.
[0083] Assume that the obtained classifier output f(x) = [0.933, 0.001, 0.044, 0.000, 0.020, 0.000, 0.002], and then get the category corresponding to the maximum value of the classifier output It is known that the true label of the image is y=1. It is found that the predicted label is consistent with the true label, which shows that the prediction is accurate.
[0084] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the incremental annotation method based on large and small model collaboration as described above. Figure 2As shown in FIG, a hardware structure diagram of any device with data processing capability in the incremental annotation method based on collaboration of large and small models provided by an embodiment of the present invention is shown, except Figure 2 In addition to the processor, memory, and network interface shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.
[0085] Accordingly, the present application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the incremental annotation method based on collaboration of large and small models as described above. The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities as described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.
[0086] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only.
[0087] It will be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.
Claims
1. An incremental annotation method based on collaboration between large and small models, characterized in that: The method comprises the following steps: Input the unlabeled image to be classified into the large model to predict a set of candidate labels; input the unlabeled image data to be classified and the set of candidate labels into the pre-trained classifier to obtain the label results; The training process of the classifier includes: For the tth round of training, in response to the unlabeled images of the current training round, the large model predicts a set of candidate labels as the initial candidate label set; Calculate the prototype based on the replayed images, classify the unlabeled images of the current training round according to the prototype, and add the classification results to the candidate label set; Calculate the temporary class prototype based on the candidate label set, calculate the similarity distribution between the unlabeled images of the current training round and the temporary class prototype to obtain the first label; calculate the second label based on the candidate label set; and use the weighted sum of the first label and the second label as the disambiguation label; The classification loss, knowledge distillation loss, and consistency regularization loss are calculated based on the disambiguation labels, unlabeled images of the current training round, and replayed images. The sum of the classification loss, knowledge distillation loss, and consistency regularization loss is used as the joint loss function, and the classifier is trained based on the joint loss function.
2. The incremental annotation method based on large and small model collaboration according to claim 1 is characterized in that: In response to the unlabeled images of the current training round, the large model predicts a set of candidate labels as the initial candidate label set. The process includes: For the tth round of training, for the data flow T is the total number of training rounds, t is the tth round of training; where each subset Contains N t samples, x i Represents the i-th sample; where sample x i The corresponding true label y i ∈Y t unknown; The large model is sample x i Predict a set of candidate labels as the initial candidate label set S i .
3. The incremental annotation method based on large and small model collaboration according to claim 1 is characterized in that: The process of calculating the prototype based on the replayed images, classifying the unlabeled images of the current training round according to the prototype, and adding the classification results to the candidate label set includes: Generate category c∈Y t The corresponding prototype μ c , the expression is as follows: Where, P c represents the sample feature set belonging to category c; φ(.) represents the feature extractor used to output sample x i The feature vector of ; j represents the jth candidate label predicted by the large model, S i Indicates that the large model responds to sample x i A set of candidate labels for prediction; Update the prototype, the expression is as follows: Where γ represents the hyperparameter; S i =S i ∪{c * } Where μ j represents the prototype corresponding to category j, c * It is the category corresponding to the sample and the prototype that is closest to it.
4. The incremental annotation method based on large and small model collaboration according to claim 1 is characterized in that: The temporary class prototype is calculated based on the candidate label set, and the similarity distribution between the unlabeled images of the current training round and the temporary class prototype is calculated. The process of obtaining the first label includes: Based on the updated candidate label set S i Calculate the temporary class prototype, the expression is as follows: P c ={φ(x i )|c∈S i } Where μ′ c Represents a temporary class prototype, P c represents the sample feature set belonging to category c; φ(.) represents the feature extractor used to output sample x i The eigenvector of Calculate sample x i The similarity distribution with the temporary class prototype, traversing each category, gets the first label; the expression is as follows: Where, Represents sample x on label j i Similarity distribution with the temporary class prototype, f(x i ) indicates that the classifier is for sample x i Output label result, μ′ j represents the temporary class prototype corresponding to label j, μ′ k Represents the temporary class prototype corresponding to label k.
5. The incremental annotation method based on large and small model collaboration according to claim 1 is characterized in that: Based on the candidate label set, the process of calculating the second label includes: Where, Indicates the second label corresponding to label j, I(j∈S i ) indicates whether the jth label exists in the updated candidate label set S i ; Update the second label based on the momentum mechanism, the expression is as follows: Where z i is a one-hot vector, f j (x i ) represents the sample x i The confidence score of label j after classifier processing, β represents the weight.
6. The incremental annotation method based on large and small model collaboration according to claim 1 is characterized in that: The process of calculating the classification loss, knowledge distillation loss, and consistency regularization loss based on the disambiguation labels, the unlabeled images of the current training round, and the replay images includes: Based on the disambiguation label p i Calculate the classification loss, the expression is as follows: Where p ij Y represents the disambiguation label corresponding to the jth label; t Represents the true label set, j represents the jth label; Calculate the knowledge distillation loss, the expression is as follows: Where, f j old (x i ) represents the sample x i The confidence score of the classifier processed on label j after t-1 rounds of training; According to the sample x i Generate a weakly enhanced view and a strongly enhanced view Calculate the consistency regularization loss to strongly enhance the view The corresponding prediction result is as close as possible to the weak enhancement view The corresponding prediction results are expressed as follows: Where, Represents a weakly enhanced view The prediction result on the jth category, Represents a strongly enhanced view The prediction result on the jth category.
7. The incremental annotation method based on large and small model collaboration according to claim 1 is characterized in that: The method further comprises: After the tth round of training, the image closest to the prototype is selected as the replay image to participate in the t+1th round of training; the expression is as follows: in Divide the entire set according to the predicted category and then select the N closest to the prototype r samples.
8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to execute the incremental annotation method based on size model collaboration as described in any one of claims 1 to 7.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the incremental annotation method based on large and small model collaboration as described in any one of claims 1 to 7.
10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the incremental annotation method based on large and small model collaboration described in any one of claims 1 to 7 is implemented.