Image classification model reuse method and system for reducing long-tail image labeling cost

Abstract

This invention discloses a method and system for reusing image classification models to reduce the cost of long-tail image annotation. It is primarily used to reduce the annotation cost of building machine learning models when image categories exhibit a long-tail distribution. The main technical solution of this invention is as follows: by acquiring a set of models related to the target task, reusing these models, and utilizing unlabeled images from the target task, performance improvement is achieved even with limited long-tail image annotations. The unlabeled image utilization step combines 1) performance evaluation of the target task model using labeled images; and 2) guidance updates of the reuse pattern from relevant models. This approach is easy to implement and highly efficient, achieving superior performance compared to existing semi-supervised learning and model reuse techniques when long-tail image annotations are insufficient.

Description

Technical Field

[0001] This invention relates to a method and system for reusing image classification models that reduces the cost of long-tail image annotation, belonging to the fields of machine learning technology and image classification technology. Background Technology

[0002] In image classification tasks, to achieve good classification results, machine learning methods typically require collecting a large number of images with clear concept annotations. In the natural world, image concepts often exhibit a long-tail distribution, meaning that a few image concept categories appear frequently, while a large number of image concept categories appear only occasionally ("long-tail images"). For classification tasks involving such images, the annotation process for long-tail images relies more heavily on expert experience and knowledge, making it difficult to respond promptly and effectively. This requires significant human and material resources, significantly increasing the cost of building image classification models.

[0003] Model reuse, by reusing related image classification models to reduce the cost of labeled images required during the training process of a target task, has attracted widespread attention in recent years. However, when long-tail image annotations are scarce for the target task, assessing the reliability of image classification model reuse becomes difficult, severely impacting its application in long-tail images. Therefore, there is an urgent need to develop classification model reuse methods that can utilize a large number of unlabeled images, thereby improving the effectiveness of image classification model reuse and the performance of the final image classification model construction when long-tail image annotations are scarce.

[0004] When image annotations are scarce, semi-supervised learning has been applied in many areas to improve the performance of image classification models by using a large number of unlabeled images. However, existing semi-supervised learning methods often rely on strong distributional assumptions when using unlabeled images, generally assuming that the frequency of concept categories in unlabeled images is roughly equal. When faced with classification tasks with long-tailed images, the proportion of concept categories in unlabeled images may be severely imbalanced, making it difficult for existing semi-supervised learning algorithms to effectively utilize unlabeled images and resulting in poor performance.

[0005] In summary, to effectively reduce the annotation cost of long-tail image classification tasks, it is urgent to develop novel semi-supervised image classification model reuse methods that can utilize a large number of unlabeled images. This would improve the effectiveness of image classification model reuse and semi-supervised learning, ultimately enhancing the classification performance of image classification models and effectively reducing annotation costs. Currently, there are relatively few technical solutions in this area, both domestically and internationally. Summary of the Invention

[0006] Objective: To address the shortcomings of existing technologies in classifying long-tailed images with poor performance, this invention provides a method and system for reusing image classification models to reduce the annotation cost of long-tailed images. It integrates the advantages of semi-supervised learning and image classification model reuse, reducing the annotation cost of long-tailed images and improving the performance of image classification models. Specifically, firstly, unlabeled images are given initial pseudo-labels by reusing relevant models. Based on this, semi-supervised learning is performed using the unlabeled images with the help of the relevant models. The error is backpropagated to the relevant models via higher-order gradients based on the performance on labeled images, optimizing the model reuse strategy, improving the effectiveness of model reuse, and ultimately enhancing the classification performance of the model.

[0007] Technical solution: A method for reusing image classification models to reduce the cost of long-tail image annotation, specifically including:

[0008] 1) Use candidate model groups to assign pseudo-labels to unlabeled images through a reuse strategy;

[0009] 2) By combining image pseudo-annotation, a semi-supervised learning image classification model is performed using annotated images to improve the model's image classification performance;

[0010] 3) Use labeled images to evaluate the model's performance, and use the second-order gradient to backpropagate the performance improvement direction to the model group, optimize the model reuse strategy, and improve the effect;

[0011] 4) Repeat the above 3 steps until the image pseudo-labels no longer change, reaching convergence, and obtain the image classification model, which can be used for image classification operations.

[0012] The steps for obtaining pseudo-labels for the unlabeled images are as follows:

[0013] Obtain unlabeled images common One sample, This represents the i-th unlabeled image sample. A candidate model group is obtained. Among them, there are a total of One model, This represents the k-th candidate model, and the candidate model group is a set of models relevant to the target task. Initialize the reuse strategy function. Strategy function The input is an unlabeled image sample. The output is a dimensional weight vector For unlabeled image samples The output of the reuse strategy is ,in Represents the image sample Individual candidate model The integration weights. Through the current reuse strategy. Predicting unlabeled images can yield a set of pseudo-labels for those images. .

[0014] In step 2), the set of pseudo-annotated images obtained through the candidate model group will be... , with the annotated image set common Semi-supervised learning is performed on labeled image samples to obtain an image classification model for the target task. .

[0015] In step 3), the steps for optimizing model reuse are as follows:

[0016] Image classification model obtained through semi-supervised learning Performance evaluation is performed on labeled images, based on the output of the contrastive classification model. With the true annotation of the image Computational model performance evaluation function The reuse strategy function is updated by feedback from the second-order gradient of the performance evaluation function. .

[0017] An image classification model reuse system that reduces the cost of long-tail image annotation includes:

[0018] Pseudo-annotation module: Uses candidate model groups to assign pseudo-annotations to unannotated images through a reuse strategy;

[0019] Image classification model learning module: Combines image pseudo-annotation and uses annotated images to perform semi-supervised learning of image classification models;

[0020] Optimize the model reuse module: Use labeled images to evaluate the model performance, and use the second gradient to backpropagate the performance improvement direction to the model group to optimize the model reuse strategy.

[0021] Looping module: The pseudo-labeling module, image classification model learning module, and optimized model reuse module are executed sequentially until the image pseudo-labeling no longer changes, reaching convergence and obtaining the image classification model.

[0022] The implementation process and methods of the system are the same and will not be described again.

[0023] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the image classification model reuse method described above for reducing the cost of long-tail image annotation.

[0024] A computer-readable storage medium storing a computer program that performs the image classification model reuse method for reducing the cost of long-tail image annotation as described above. Attached Figure Description

[0025] Figure 1 This is a flowchart of the method of the present invention;

[0026] Figure 2 Flowchart for obtaining pseudo-annotations through reuse strategy

[0027] Figure 3 This is a flowchart of a semi-supervised learning process that utilizes unlabeled images through pseudo-labeling. Detailed Implementation

[0028] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0029] like Figure 1 The flowchart shown is for an image classification model reuse method that reduces the cost of long-tail image annotation. Step 1 is the initial action.

[0030] Step 2: Through candidate model groups and reuse strategy function The unlabeled image was calculated. pseudo-annotated image set .

[0031] Obtain unlabeled images common One sample, This represents the i-th unlabeled image sample. A candidate model group is obtained. Among them, there are a total of One model, This is the k-th candidate model. Initialize the reuse strategy function. Strategy function The input is an unlabeled image sample. The output is a dimensional weight vector For unlabeled image samples The output of the reuse strategy is ,in Represents the image sample Individual candidate model The integration weights. Through the current reuse strategy. Predicting unlabeled images can yield a set of pseudo-labels for those images. .

[0032] Step 3 uses a set of pseudo-annotated images and labeled image set Semi-supervised learning is used to obtain the image classification model for the current task. The set of pseudo-annotated images Includes unlabeled images and its corresponding pseudo-labels , .

[0033] Step 4: Utilize the labeled image set Evaluate the image classification model for the current task Update the reuse strategy function based on error feedback. Specifically, using a set of labeled images. In Compare the model output with each image sample. With image real annotation The difference in the current target model The formula for performance evaluation is:

[0034]

[0035] The evaluation function The classic cross-entropy loss function is used. Then, the chain rule is applied to obtain... For reuse strategy functions The gradient of is given by the formula:

[0036]

[0037] Through the second gradient The reuse strategy function can be obtained. The direction of parameter updates.

[0038] Step 5: Determine the model Whether it converges, when the model In the image collection When the performance on the current model no longer changes, the current model will be... This is the final model output; otherwise, return to step 2 and use the updated policy function. Continue learning.

[0039] Figure 2 yes Figure 1 The specific workflow of step 2.

[0040] The process begins with step 20.

[0041] In step 21, obtain the set of unlabeled images. In step 22, a set of historical models related to the target task are obtained as candidate models. These historical models were trained separately on large sets of labeled samples from related tasks. Since the data distribution of related tasks is usually not exactly the same as that of the current image classification target task, these historical models often perform poorly when directly applied to the target task.

[0042] In step 23, a reuse strategy model is initialized. Its function input is an image sample. The output is a multiplexed one. dimensional vector For the sample The output of the reuse strategy is ,in Representative for the sample Individual model The integrated weights. After initialization, the reuse strategy... For any sample Output with uniform distribution That is, the individual models in the model group are used uniformly.

[0043] In step 24, the policy function is used. Get the set of unlabeled images The pseudo-labels are merged to obtain a set of unlabeled images. pseudo-annotated image set .

[0044] Figure 3 yes Figure 1 The specific workflow of step 3.

[0045] The process begins from step 30.

[0046] In step 31, the image set required for semi-supervised learning is obtained. and .

[0047] In step 32, the classic cross-entropy classification loss function is used. Calculate the target model In the image collection middle Annotated images and middle The empirical errors on unlabeled images are summed. Target model The loss function formula is:

[0048]

[0049] Then, the gradient is calculated using the error. Update model The parameters are adjusted until the model converges.

[0050] An image classification model reuse system that reduces the cost of long-tail image annotation includes:

[0051] Pseudo-annotation module: Uses candidate model groups to assign pseudo-annotations to unannotated images through a reuse strategy;

[0052] Image classification model learning module: Combines image pseudo-annotation and uses annotated images to perform semi-supervised learning of image classification models;

[0053] Optimize the model reuse module: Use labeled images to evaluate the model performance, and use the second gradient to backpropagate the performance improvement direction to the model group to optimize the model reuse strategy.

[0054] Looping module: The pseudo-labeling module, image classification model learning module, and optimized model reuse module are executed sequentially until the image pseudo-labeling no longer changes, reaching convergence and obtaining the image classification model.

[0055] Obviously, those skilled in the art should understand that the steps of the image classification model reuse method for reducing the cost of long-tail image annotation in the above-described embodiments of the present invention, or the modules of the image classification model reuse system for reducing the cost of long-tail image annotation, can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

[0056] This invention addresses the performance limitations of traditional machine learning methods when long-tailed images have scarce annotations by proposing a method for reusing image classification models. On one hand, by mining unlabeled images, it improves the ability to evaluate the performance of model sets during model reuse, thereby increasing the utilization efficiency of historical models. On the other hand, by reusing model sets to utilize unlabeled images, it reduces the method's dependence on image annotations and improves the utilization efficiency of unlabeled images. To further illustrate the effectiveness of the technique, Table 1 presents experimental results on long-tailed images.

[0057] This invention uses an image classification benchmark dataset to verify its effectiveness. The experiment classifies the presence of seven animal traits. For each classification setting, animal samples from different habitats were first collected, and models based on extensive labeled images were built for four habitats: plains, mountains, streams, and forests, constructing model groups. Then, the experiment randomly sampled image sets from all samples in any habitat. and Using the target image as the objective, comparative experiments were conducted between this invention and benchmark model reuse methods and benchmark semi-supervised methods. The experimental results report the classification accuracy (mean ± standard deviation) of five randomly repeated experiments.

[0058] Table 1 Classification accuracy results (the higher the value, the better the performance).

[0059]

Claims

1. An image classification model reuse method for reducing long-tail image annotation cost, characterized in that, Specifically, it includes: 1) Use candidate model groups to assign pseudo-labels to unlabeled images through a reuse strategy; The steps for obtaining pseudo-labels for the unlabeled images are as follows: Obtain unlabeled images common One sample, Represent the i-th unlabeled image sample; obtain the candidate model group. Among them, there are a total of One model, The k-th candidate model is defined as a group of models related to the target task, which are trained from labeled sample sets of related tasks. The reuse strategy function is initialized. Strategy function The input is an unlabeled image sample. The output is a dimensional weight vector For unlabeled image samples The output of the reuse strategy is ,in Represents the image sample Individual candidate model Integration weights; through the current reuse strategy For unlabeled image sets Each image sample in Making predictions can yield a set of pseudo-labels for unlabeled images. ; 2) Combine image pseudo-annotations with labeled images to perform semi-supervised learning of an image classification model; the set of pseudo-annotated images obtained through the candidate model group... , with the annotated image set common Semi-supervised learning is performed on labeled image samples to obtain an image classification model for the target task. ; 3) Evaluate the model's performance using labeled images, and backpropagate the performance improvement direction to the model group using the second-order gradient to optimize the model reuse strategy; the steps to optimize model reuse are as follows: Image classification model obtained through semi-supervised learning Performance evaluation is performed on the labeled image, and the performance evaluation function is calculated. The reuse strategy function is updated by feedback from the second-order gradient of the performance evaluation function. ; Using labeled image sets Evaluate the image classification model for the current task Update the reuse strategy function based on error feedback. Specifically, using a set of labeled images In Compare the model output with each image sample. With image real annotation The difference in the current target model The formula for performance evaluation is: The evaluation function The cross-entropy loss function is used, and then the result is obtained through the chain rule. For reuse strategy functions The gradient of is given by the formula: Through the second gradient Obtain the reuse strategy function The direction of parameter updates; Judgment Model Whether it converges, when the model In the image collection When the performance on the current model no longer changes, the current model will be... As the final model output, otherwise use the updated policy function. Continue learning; Using cross-entropy classification loss function Calculate the target model In the image collection middle Annotated images and middle The empirical errors on unlabeled images are summed; the target model The loss function formula is: A collection of pseudo-annotated images; 4) Repeat steps 1)-3) above until the image pseudo-labels no longer change, reaching convergence.

2. A system for implementing the image classification model reuse method for reducing the cost of long-tail image annotation as described in claim 1, characterized in that, include: Pseudo-annotation module: Uses candidate model groups to assign pseudo-annotations to unannotated images through a reuse strategy; Image classification model learning module: Combines image pseudo-annotation and uses annotated images to perform semi-supervised learning of image classification models; Optimize the model reuse module: Use labeled images to evaluate the model performance, and use the second gradient to backpropagate the performance improvement direction to the model group to optimize the model reuse strategy. Looping module: The pseudo-labeling module, image classification model learning module, and optimized model reuse module are executed sequentially until the image pseudo-labeling no longer changes, reaching convergence and obtaining the image classification model.

3. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image classification model reuse method for reducing the cost of long-tail image annotation as described in claim 1.

4. A computer-readable storage medium storing a computer program that performs the image classification model reuse method for reducing the cost of long-tail image annotation as described in claim 1.

Images