Pre-training model fine-tuning method based on generated data distribution alignment
Patent Information
- Application Number
- CN202311486070.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-09
- Publication Date
- 2026-11-03
- Estimated Expiration
- 2043-11-09
AI Technical Summary
Large-scale pre-trained models consume a lot of computational resources and time when transferred to downstream tasks, and are prone to catastrophic forgetting and overfitting. In particular, when there are few training samples, traditional data augmentation methods cannot effectively improve the model's generalization ability.
Generated data is introduced as a third modality and trained together with real and text data. Matching between images, text and generated data is achieved through cue engineering. A pre-trained model tuning method that aligns the distribution of generated data is used to reduce dependence on training samples and overfitting.
While reducing computational resources and time costs, it improves the model's generalization ability in downstream tasks, avoids catastrophic forgetting and overfitting, and improves image recognition performance.
Smart Images
Figure CN117422954B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision, and specifically relates to a method for tuning a pre-trained model based on the alignment of generated data distribution. Background Technology
[0002] In recent years, large-scale pre-trained models have demonstrated strong generalization capabilities across various downstream tasks. These models utilize vast amounts of visual and textual data collected online, trained using a contrastive learning approach, ultimately resulting in models that can be transferred to different downstream tasks and achieve good performance. However, the sheer size of these models means that transferring them to downstream tasks consumes enormous computational resources and time, and the transfer process can also lead to catastrophic forgetting.
[0003] To reduce computational resource requirements while preserving the generalization ability of pre-trained models, cue-based optimization engineering has emerged. This method freezes the core of the pre-trained model and selects to add relatively fewer parameters as cues for training, thus preserving the pre-trained model and reducing computational resources. Even so, overfitting to the training data during training can still easily lead to a loss of generalization ability. In particular, when the training samples are limited, the dependence of the training transfer process for downstream tasks on the samples increases dramatically. Data augmentation can be used to address this challenge.
[0004] Traditional data augmentation involves preprocessing the original training data using various methods to obtain multiple, slightly different images for use. While this approach extracts more information from the training data through more efficient utilization, it doesn't provide the network with any information beyond the training data itself, thus limiting the scope of the information obtained. Furthermore, this excessive manipulation of the training data can exacerbate overfitting in the model. Summary of the Invention
[0005] To address the aforementioned issues, this invention aims to provide a pre-trained model tuning method based on the alignment of generated data distribution. This method provides highly diverse and high-quality information to the pre-trained model, reducing reliance on training samples and overfitting during training. Image generation tasks can generate new sample data to alleviate the drawbacks of insufficient sample data in few-shot learning. With the improvement of generated image quality, image generation tasks have gradually become a reliable aid in addressing few-shot learning. Considering the domain offset problem between generated data and real data, this invention introduces generated data as a new domain independent of real data and text data, and uses cue engineering as a means to achieve matching between the three domains, thereby reliably utilizing the generated data.
[0006] To achieve the above objectives, the technical solution of the present invention is as follows:
[0007] A pre-trained model tuning method based on generated data distribution alignment includes the following steps:
[0008] S1. Obtain real image label text, real image dataset, and generated image dataset with the same category as the real image dataset for training the pre-trained model in the downstream task;
[0009] S2. Train the pre-trained model using real image label text, real image data, and generated image data. Based on the pre-trained model, the text encoder and image encoder encode the real image label text, real image data, and generated image data to obtain text features, real image features, and generated image features. The optimization objective is to minimize the sum of the cross-entropy optimization loss between real image features and text features, the cross-entropy optimization loss between generated image features and text features, and the Kullback-Leibler divergence loss between real image features and generated image features.
[0010] S3. Train the pre-trained model based on the optimization objective, and use the trained model to output the prediction results for the given image classification task.
[0011] Furthermore, in step S1, the real image label text is represented as {text1, text2, ..., text...} C}, where text i Let C represent the name of the i-th category in English text form, and C represent the total number of categories; the real image dataset is represented as {(x1,y1),(x2,y2),...,(x...} N ,y N )}, where N is the total number of images in the real dataset, x i and y i Let y represent the i-th image and its corresponding class label, respectively. i ∈{1,...,C}, each class contains multiple images; the generated image dataset is Where M is the total number of images in the generated dataset. and These represent the i-th image and its corresponding class label, respectively. Each class contains n images, where
[0012] Furthermore, in step S2, obtaining text features specifically includes:
[0013] S21. Construct phrases that better match the image description for the real image label text. Search for the order of each word in the phrase in a given dictionary and replace the words in the phrase with integer numbers based on the word order.
[0014] S22. Encode the numbers in each phrase to map them to a higher dimension, resulting in a C*N size. seq *d text A tensor, where C represents the total number of categories, N seq d represents the upper limit of the number of words in a predefined phrase. text The dimension encoded for each number;
[0015] S23. Input the tensor after encoding the text of the real image labels into the text encoder of the pre-trained model to obtain the encoded text features. in It is the text feature encoded by the text encoder for the label text of the i-th category, with a size of 1*D, where D is the feature dimension.
[0016] Furthermore, in step S2, obtaining the real image features and the generated image features specifically includes:
[0017] S24. When training the pre-trained model, a batch processing method is used to randomly sample a batch of real image samples from the real image dataset. Where the batch size N bs Pre-given; based on sampled real image sample B real Sample the generated image data to obtain generated image samples of the same batch size.
[0018] S25, Take real image sample B real Encoded image features are obtained from the image encoder input into the pre-trained model. It is the image feature encoded by the image encoder for the i-th real image, with a size of 1*D; the generated image sample B fake An encoder with the same image encoder structure as the pre-trained model but different parameters produces encoded features for the generated image. It is the image feature encoded by the image encoder for the i-th generated image, with a size of 1*D.
[0019] Furthermore, in step S2, determining the optimization objective specifically includes:
[0020] S26. Transfer the real image features F image_real With text features F text The transpose of the product is used to obtain the similarity between each image in the real image and each category of text, which is then used as the prediction result. It is the prediction result of the pre-trained model for the i-th real image, with a size of 1*C;
[0021] S27. Generate image features F image_fake With text features F text The transpose of the multiplication is used to obtain the similarity between each image in the generated image and each category of text, which is then used as the prediction result. It is the prediction result of the pre-trained model for the i-th generated image, with a size of 1*C;
[0022] S28. Use the Kullback-Leibler divergence loss function L between the generated image prediction results and the ground image prediction results. KL Simultaneously, the prediction results of a batch of real images are used to construct the cross-entropy optimization loss function L. CE_real The cross-entropy optimization loss function L is composed of a batch of generated image prediction results. CE_fake ;
[0023] S29. Based on the cross-entropy optimized loss function and the Kullback-Leibler divergence loss function, the total loss function is obtained as follows:
[0024] L = L KL +α*L CE_real +β*L CE_fake
[0025] Where α and β are pre-set weight parameters used to balance the contributions of the two loss functions.
[0026] Furthermore, in step S28, the Kullback-Leibler divergence loss function is:
[0027]
[0028] in, Where τ is the temperature coefficient.
[0029] Furthermore, in step S28, the cross-entropy optimization loss function is:
[0030]
[0031] Among them, y ij z is the true label of the image. ij Output the prediction results for the model.
[0032] Furthermore, in step S3, training the pre-trained model based on the optimization objective includes:
[0033] S31. Based on the obtained total loss function, use the Adam optimizer or the stochastic gradient descent optimizer with momentum, and the backpropagation algorithm to train the pre-trained model.
[0034] S32. Repeat steps S23 to S31 until the model converges.
[0035] Furthermore, in step S3, the prediction results for the given image classification task are output using the trained model, including:
[0036] x i Input the trained model and obtain the predicted output z. i The predicted label is:
[0037]
[0038] This is the prediction result for the image.
[0039] The pre-trained model tuning method based on generated data distribution alignment of the present invention has the following beneficial effects:
[0040] First, this invention utilizes cueing engineering for training, efficiently transferring large-scale pre-trained models to downstream tasks while reducing computational resource and time costs, thus achieving significant performance gains. Furthermore, cueing engineering does not alter the core framework of the large-scale pre-trained model during training, preserving its generalization capabilities across various tasks or domains.
[0041] Secondly, this invention follows the original idea of pre-trained models, cleverly introducing generated data as a data augmentation method. While pre-trained models achieve matching between image and text modalities through contrastive learning during pre-training, this invention uses newly introduced generated data as a third modality, enabling further matching between the generated image, real image, and text modalities, thus avoiding the catastrophic forgetting phenomenon of pre-trained knowledge.
[0042] In addition, compared with traditional data augmentation methods, the data augmentation method proposed in this invention introduces new generated data, which increases the diversity of training data while reducing the dependence on the original training data, thereby achieving multiple purposes such as improving the model's generalization ability and avoiding overfitting.
[0043] The pre-trained model tuning method based on generated data distribution alignment of this invention has significant application value in computer vision and few-shot training scenarios. The cueing engineering used in this invention can be lightweightly inserted into pre-trained models with different structures to achieve different task objectives. The encoder required for generating data can also be selected and used according to different pre-trained models, thereby improving the performance of different tasks. Attached Figure Description
[0044] Figure 1 This is a flowchart illustrating the pre-trained model tuning method based on generated data distribution alignment according to the present invention. Detailed Implementation
[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0046] Conversely, this invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of the invention as defined in the claims. Furthermore, to provide a better understanding of the invention, certain specific details are described in detail below. However, those skilled in the art will fully understand the invention even without these detailed descriptions.
[0047] refer to Figure 1 In a preferred embodiment of the present invention, the pre-trained model tuning method based on generated data distribution alignment includes the following steps:
[0048] First, obtain the real image label text, the real image dataset, and the generated image dataset that are consistent with the categories of the real image dataset for training the pre-trained model in the downstream task.
[0049] In this embodiment, the real image label text is represented as {text1,text2,…,text...} C}, where text i Let C represent the name of the i-th category in English text form, and C represent the total number of categories.
[0050] The real image dataset is represented as {(x1,y1),(x2,y2),…,(x N ,y N )}, where N is the total number of images in the real dataset, x i and y i Let y represent the i-th image and its corresponding class label, respectively. i ∈{1,...,C}, each class contains multiple images.
[0051] The generated image dataset is represented as Where M is the total number of images in the generated dataset. and These represent the i-th image and its corresponding class label, respectively. Each class contains n images, where
[0052] Secondly, the pre-trained model is trained using real image label text, real image data, and generated image data. Based on the pre-trained model's text encoder and image encoder, text, real image data, and generated image data are encoded to obtain text features, real image features, and generated image features. The optimization objective is determined to be minimizing the sum of the cross-entropy optimization loss between real image features and text features, the cross-entropy optimization loss between generated image features and text features, and the Kullback-Leibler divergence loss between real image features and generated image features. The pre-trained model is a pre-trained visual language model, such as CLIP or ALIGN. Specifically, the steps include:
[0053] The first step involves constructing phrases that better match the image description for the real image label text. Then, within a given dictionary, the order of each word in the phrase is searched, and words in the phrase are replaced with integers based on this order. Finally, the numbers in each phrase are encoded to map to a higher dimension, resulting in a C*N value. seq *d text A tensor, where C represents the total number of categories, N seq d represents the upper limit of the number of words in a predefined phrase. text The dimension after encoding each number.
[0054] The second step involves batch processing when training the pre-trained model. First, a batch of real image samples is randomly sampled from the real image dataset. Where the batch size N bs Pre-given; based on sampled real image sample B real Sample the generated data to obtain generated image samples of the same batch size.
[0055] The third step is to input the tensor of the real image label text encoding into the text encoder of the pre-trained model to obtain the encoded text features. This is the text feature encoded by the text encoder for the label text of the i-th category, with a size of 1*D, where D is the feature dimension. The real image sample B... real Encoded image features are obtained from the image encoder input into the pre-trained model. This is the image feature encoded by the image encoder for the i-th real image, with a size of 1*D. The generated image sample B... fake An encoder with the same image encoder structure as the pre-trained model but different parameters produces encoded image features. It is the image feature encoded by the image encoder for the i-th generated image, with a size of 1*D.
[0056] The fourth step is to extract the features of the real image F. image_real With text features F text The transpose of the product is used to obtain the similarity between each image in the real image and each category of text, which is then used as the prediction result. This is the model's prediction result for the i-th real image, with a size of 1*C. The generated image features F... image_fake With text features F text The transpose of the multiplication is used to obtain the similarity between each image in the generated image and each category of text, which is then used as the prediction result. It is the model's prediction result for the i-th generated image, with a size of 1*C.
[0057] The fifth step involves using the Kullback-Leibler divergence loss function L between the generated image prediction and the ground image prediction. KL Simultaneously, a batch of real image predictions is used to construct the cross-entropy optimization loss function L. CE_real A batch of generated image predictions constitutes the cross-entropy optimization loss function L. CE_fake .
[0058] In this embodiment, the Kullback-Leibler divergence loss function is:
[0059]
[0060] in, Where τ is the temperature coefficient.
[0061] The cross-entropy optimization loss function is:
[0062]
[0063] Among them, y ij z is the true label of the image. ij Output the prediction results for the model.
[0064] Step 6: Based on the cross-entropy optimization loss and the Kullback-Leibler divergence loss, the total loss function is obtained as follows:
[0065] L = L KL +α*L CE_real +β*L CE_fake
[0066] Where α and β are pre-set weight parameters used to balance the contributions of the two loss functions.
[0067] The seventh step is to train the pre-trained model using the Adam optimizer or a stochastic gradient descent optimizer with momentum, and the backpropagation algorithm, based on the obtained total loss function.
[0068] Step 8: Repeat steps 2 through 7 until the model converges.
[0069] Finally, the trained model is used to output the prediction result for a given image classification task. Specifically, the method for calculating the classification prediction result for a given image is as follows: x i Inputting the trained deep neural network yields the predicted output z. i The predicted label is:
[0070]
[0071] This is the prediction result for the image.
[0072] Through the above technical solutions, this invention provides a pre-trained model optimization method based on generated data distribution alignment. This invention introduces generated data as a modality in addition to real data and text data, and achieves matching between the three modalities through cue engineering. This enhances the generalization ability of the pre-trained model on downstream tasks, alleviates overfitting during model training, and improves image recognition performance.
[0073] It will be apparent to those skilled in the art that the modules or steps of this invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this invention is not limited to any particular hardware and software combination.
[0074] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A pre-trained model tuning method based on generated data distribution alignment, characterized in that, Includes the following steps: S1. Obtain real image label text, real image dataset, and generated image dataset with the same category as the real image dataset for training the pre-trained model in the downstream task; S2. Train the pre-trained model using real image label text, real image data, and generated image data. Based on the pre-trained model, the text encoder and image encoder encode the real image label text, real image data, and generated image data to obtain text features, real image features, and generated image features. The optimization objective is to minimize the sum of the cross-entropy optimization loss between real image features and text features, the cross-entropy optimization loss between generated image features and text features, and the Kullback-Leibler divergence loss between real image features and generated image features. S3. Train the pre-trained model based on the optimization objective, and use the trained model to output the prediction results for the given image classification task.
2. The pre-trained model tuning method based on generated data distribution alignment as described in claim 1, characterized in that, In step S1, the real image label text is represented as {text1, text2, ..., text...} C }, where text i Let C represent the name of the i-th category in English text form, and C represent the total number of categories; the real image dataset is represented as {(x1,y1),(x2,y2),...,(x...} N ,y N )}, where N is the total number of images in the real dataset, x i and y i Let y represent the i-th image and its corresponding class label, respectively. i ∈{1,...,C}, each class contains multiple images; the generated image dataset is Where M is the total number of images in the generated dataset. and These represent the i-th image and its corresponding class label, respectively. Each class contains n images, where 3. The pre-trained model tuning method based on generated data distribution alignment as described in claim 2, characterized in that, In step S2, obtaining text features specifically includes: S21. Construct phrases that better match the image description for the real image label text. Search for the order of each word in the phrase in a given dictionary and replace the words in the phrase with integer numbers based on the word order. S22. Encode the numbers in each phrase to map them to a higher dimension, resulting in a C*N size. seq *d text A tensor, where C represents the total number of categories, N seq d represents the upper limit of the number of words in a predefined phrase. text The dimension encoded for each number; S23. Input the tensor after encoding the text of the real image labels into the text encoder of the pre-trained model to obtain the encoded text features. in It is the text feature encoded by the text encoder for the label text of the i-th category, with a size of 1*D, where D is the feature dimension.
4. The pre-trained model tuning method based on generated data distribution alignment as described in claim 3, characterized in that, In step S2, obtaining the real image features and the generated image features specifically includes: S24. When training the pre-trained model, a batch processing method is used to randomly sample a batch of real image samples from the real image dataset. Where the batch size N bs Pre-given; based on sampled real image sample B real Sample the generated image data to obtain generated image samples of the same batch size. S25, Take real image sample B real Encoded image features are obtained from the image encoder input into the pre-trained model. It is the image feature encoded by the image encoder for the i-th real image, with a size of 1*D; the generated image sample B fake An encoder with the same image encoder structure as the pre-trained model but different parameters produces encoded features for the generated image. It is the image feature encoded by the image encoder for the i-th generated image, with a size of 1*D.
5. The pre-trained model tuning method based on generated data distribution alignment as described in claim 4, characterized in that, In step S2, determining the optimization objective specifically includes: S26. Transfer the real image features F image_real With text features F text The transpose of the product is used to obtain the similarity between each image in the real image and each category of text, which is then used as the prediction result. It is the prediction result of the pre-trained model for the i-th real image, with a size of 1*C; S27. Generate image features F image_fake With text features F text The transpose of the multiplication is used to obtain the similarity between each image in the generated image and each category of text, which is then used as the prediction result. It is the prediction result of the pre-trained model for the i-th generated image, with a size of 1*C; S28. Use the Kullback-Leibler divergence loss function L between the generated image prediction results and the ground image prediction results. KL Simultaneously, the prediction results of a batch of real images are used to construct the cross-entropy optimization loss function L. CE_real The cross-entropy optimization loss function L is composed of a batch of generated image prediction results. CE_fake ; S29. Based on the cross-entropy optimized loss function and the Kullback-Leibler divergence loss function, the total loss function is obtained as follows: L=L KL +a*L CE_real +β*L CE_fake Where α and β are pre-set weight parameters used to balance the contributions of the two loss functions.
6. The pre-trained model tuning method based on generated data distribution alignment as described in claim 5, characterized in that, In step S28, the Kullback-Leibler divergence loss function is: in, Where τ is the temperature coefficient.
7. The pre-trained model tuning method based on generated data distribution alignment as described in claim 5, characterized in that, In step S28, the cross-entropy optimization loss function is: Among them, y ij z is the true label of the image. ij Output the prediction results for the model.
8. The pre-trained model tuning method based on generated data distribution alignment as described in any one of claims 5-7, characterized in that, Step S3, training the pre-trained model based on the optimization objective, includes: S31. Based on the obtained total loss function, use the Adam optimizer or the stochastic gradient descent optimizer with momentum, and the backpropagation algorithm to train the pre-trained model. S32. Repeat steps S23 to S31 until the model converges.
9. The pre-trained model tuning method based on generated data distribution alignment as described in claim 8, characterized in that, In step S3, the prediction results for the given image classification task are output using the trained model, including: x i Input the trained model and obtain the predicted output z. i The predicted label is: This is the prediction result for the image.