Fine tuning optimization method for improving fine granularity discrimination capability of multi-modal model

Abstract

The invention relates to a fine tuning optimization method for improving the fine granularity discrimination capability of a multi-modal model, and belongs to the field of artificial intelligence. Angle boundary auxiliary loss of ArcFace is introduced into an image encoder channel in a fine tuning stage to optimize joint representation learning of images and texts. The method comprises a pre-training stage and a fine tuning stage, the pre-training stage is used for training on a large-scale image-text pair data set, and the fine tuning stage is used for adjusting model parameters by fixing all parameters of a text branch and most parameters of an image branch on the basis of a specific field data set and combining an auxiliary loss function on the image branch. Therefore, optimization and improvement of the specific fine particle scene discrimination capability of the multi-modal model are realized.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and specifically relates to a fine-tuning optimization method for improving the fine-grained discrimination capability of a multimodal model. Background Art

[0002] Based on public large-scale data sets, multimodal models often have the phenomenon of having more general knowledge than specialized scene segmentation capabilities. As a typical representative of text vision multimodal models, the CLIP model uses a contrastive learning framework to pre-train with images and text pairs of general concepts (people, cars), but it has the problem of insufficient adaptability to specific fine-grained tasks (such as distinguishing specific car models, etc.). The ArcFace model enhances the feature separation of different faces in the field of face recognition by introducing angular boundary loss, which can provide high-resolution fine discrimination. Its method can be used in multimodal learning to improve feature discrimination capabilities. Summary of the invention

[0003] 1. Technical issues to be resolved

[0004] The technical problem to be solved by the present invention is how to provide a fine-tuning optimization method for improving the fine-grained discrimination ability of the multimodal model, so as to solve the problem that the multimodal model often has sufficient general ability but insufficient professional segmentation scenario ability.

[0005] (II) Technical solution

[0006] In order to solve the above technical problems, the present invention proposes a fine-tuning optimization method for improving the fine-grained discrimination ability of a multimodal model, the method comprising the following steps:

[0007] Step S1, pre-training the CLIP multimodal model to obtain trained weights and network structure;

[0008] Step S2, constructing a task-specific image-text pair dataset;

[0009] Step S3: Fine-tuning training

[0010] The preloaded weights of the CLIP multimodal model are the weights of step S1. During fine-tuning training, the images in the image-text pair dataset of the specific task are input into the image encoder of the CLIP multimodal model, and the texts are input into the text encoder of the CLIP multimodal model.

[0011] The image branch passes through the image encoder, fully connected layer, and normalization layer to obtain the image feature vector, which is input into the category center layer to calculate the cosine similarity between the image encoding vector and the category center vector, and obtains the cosine similarity after applying the angle boundary in the angle boundary loss calculation layer, thereby obtaining the angle boundary loss.

[0012] The text branch passes through the text encoder, fully connected layer, and normalization layer to obtain the text feature vector, which is used together with the image feature vector output by the normalization layer of the image branch to calculate the contrast loss of the text-image pair;

[0013] In the weighted loss layer, the contrast loss and angle boundary loss of the text-image pair are weighted to get the total loss;

[0014] Step S4: Reasoning application

[0015] The fine-tuned trained model weights are saved according to the standard CLIP multimodal model network structure, and then deployed to practical applications according to the standard CLIP mode.

[0016] (III) Beneficial effects

[0017] The present invention proposes a fine-tuning optimization method for improving the fine-grained discrimination capability of a multimodal model. The present invention has the following beneficial effects:

[0018] 1. Angle boundary loss is introduced in the CLIP model:

[0019] The angular boundary loss in ArcFace is introduced into the image encoder branch to enhance the category separation of the image encoder in the feature space. With the help of the angular boundary loss, the model's ability to discriminate granularity tasks in specific fields is improved.

[0020] 2. Combination of contrastive learning and angle boundaries:

[0021] By combining contrastive learning with angular boundary loss, a new joint multi-task loss function is proposed, which enables the model to maintain general knowledge while maintaining fine-grained distinction of domain data during domain data migration and fine-tuning, while reducing the amount of data and computing power requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flow chart of the pre-training method in the prior art;

[0023] Figure 2 It is a flow chart of the fine-tuning training method of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0025] The present invention belongs to the field of artificial intelligence, and in particular to the technology of using natural language supervision to improve the performance of computer vision models. Specifically, the present invention provides a method of integrating contrastive learning and angular boundary loss, which is used to optimize and improve the accuracy of image coding representation learning, and is suitable for fine-tuning optimization on specific field data sets.

[0026] In view of the above problems, the present invention provides an improved CLIP model, which optimizes the joint representation learning of images and texts by introducing the ArcFace angle boundary auxiliary loss to the image encoder channel in the fine-tuning stage. The method includes two stages: pre-training and fine-tuning. The pre-training stage uses training on a large-scale image-text pair dataset. The fine-tuning stage is based on a specific domain dataset by fixing all parameters of the text branch and most parameters of the image branch, and adjusting the model parameters in the image branch in combination with an auxiliary loss function, thereby optimizing and improving the multimodal model's ability to discriminate specific fine-grained scenes.

[0027] The complete process of the present invention consists of two parts, namely pre-training and fine-tuning training. The innovation of the present invention is carried out in the fine-tuning stage.

[0028] The present invention is based on the mainstream CLIP multimodal model framework to construct an image encoder and a text encoder. The image encoder is built based on Vision Transformer to extract the feature representation of the image. The text encoder is built based on Transformer to extract the feature representation of the text.

[0029] The two-part flowchart is as follows Figure 1 , 2 As shown, the image feature vector is given by Represented by, the text feature vector is Indicates that the category center layer parameters are given by express.

[0030] The present invention provides a fine-tuning optimization method for improving the fine-grained discrimination capability of a multimodal model, the method comprising the following steps:

[0031] Step S1: Pre-train the CLIP multimodal model to obtain trained weights and network structure

[0032] Use the publicly available pre-trained CLIP multimodal model weights and network structure, corresponding to Figure 1 You can also train it on a large-scale image-text pair dataset to obtain the pre-trained CLIP multimodal model weights and network structure.

[0033] Step S2: Build a task-specific image-text pair dataset

[0034] Collect and construct a task-specific image-text pair dataset, where each pair contains an image and a series of associated text labels. i ,text i}, i = 1 ~ M, M is the total number of sample pairs, assuming there are C categories. The dataset should cover a variety of categories and scenarios to ensure that the model can learn generalized feature representations.

[0035] Step S3: Fine-tuning training

[0036] The preloaded weights of the CLIP multimodal model are the weights of step S1, and the network structure of fine-tuning training is shown in Figure 2 , the adjusted part is the red part;

[0037] like Figure 2 As shown, the image in the image-text pair dataset for a specific task of the present invention is input into the image encoder of the CLIP multimodal model, and the text is input into the text encoder of the CLIP multimodal model;

[0038] The image branch passes through the image encoder, fully connected layer, and normalization layer to obtain the image feature vector, which is input into the category center layer to calculate the cosine similarity between the image encoding vector and the category center vector, and obtains the cosine similarity after applying the angle boundary in the angle boundary loss calculation layer, thereby obtaining the angle boundary loss.

[0039] The text branch passes through the text encoder, fully connected layer, and normalization layer to obtain the text feature vector, which is used together with the image feature vector output by the normalization layer of the image branch to calculate the contrast loss of the text-image pair;

[0040] In the weighted loss layer, the contrast loss and angular boundary loss of the text-image pair are weighted to obtain the total loss.

[0041] The main adjustment content of the present invention is the optimization of the loss function, which is specifically introduced as follows:

[0042] Loss function adjustment: Combine contrastive learning loss and angular boundary loss to construct a new loss function for training image encoders and text encoders, so that matching image-text pairs are closer in feature space and mismatched pairs are farther away, while maintaining the original capabilities of the pre-trained model. Based on the pre-trained model, fine-tune using the selected dataset and loss function to adjust the model parameters to suit the specific task. The new loss function is as follows:

[0043] L total =(1-λ)·L CLIP +λ·L Arc

[0044] λ is a hyperparameter, set to 0.4 in this case, which is used to control the impact of Arc loss on the total loss.

[0045] The contrastive learning loss of the text-image pair is calculated as follows, which is the loss function of the standard CLIP algorithm:

[0046]

[0047] ·v i and t i They are the outputs of the image and text after passing through their respective encoders.

[0048] · and They are v i and t i The normalized result.

[0049] ·sim(a,b) is the similarity between vectors.

[0050] τ is a temperature hyperparameter that controls the degree of distinguishability.

[0051] N is the batch size.

[0052] In order to increase the discrimination of image feature encoding, an auxiliary angle boundary loss L based on angle boundary is added to the image branch. Arc , we first calculate the cosine similarity between the normalized image vector and the category center. The category center vector is d is the length of the normalized vector, C is the number of categories, and each column is a category center vector w j , j ranges from 1 to C, corresponding to the number of categories. Angular boundary loss L Arc The calculation steps are as follows:

[0053] S31, Normalization: Image encoding vector v i and the category center vector w j All are normalized:

[0054]

[0055] S32, cosine similarity: Calculate the normalized cosine similarity:

[0056]

[0057] S33, cosine similarity after applying angle boundary:

[0058] Apply an angular margin m to the similarity of the correct category:

[0059]

[0060] where y i Yes i The corresponding true category number, y i The value range is an integer from 1 to C.

[0061] S34, the final angle boundary loss calculation is as follows:

[0062]

[0063] Where N is the batch size and s is a hyperparameter, which is a scaling factor used to amplify feature normalization.

[0064] Step S4: Reasoning application

[0065] The fine-tuned trained model weights are saved according to the standard CLIP multimodal model network structure, and then deployed to practical applications such as image retrieval systems and automatic annotation tools according to the standard CLIP mode.

[0066] The present invention has the following beneficial effects:

[0067] 1. Introducing angle boundary loss in CLIP multimodal model:

[0068] The angular boundary loss in ArcFace is introduced into the image encoder branch to enhance the category separation of the image encoder in the feature space. With the help of the angular boundary loss, the model's ability to discriminate granularity tasks in specific fields is improved.

[0069] 2. Combination of contrastive learning and angle boundaries:

[0070] By combining contrastive learning with angular boundary loss, a new joint multi-task loss function is proposed, which enables the model to maintain general knowledge while maintaining fine-grained distinction of domain data during domain data migration and fine-tuning, while reducing the amount of data and computing power requirements.

[0071] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A fine-tuning optimization method for improving the fine-grained discrimination capability of a multimodal model, characterized in that: The method comprises the following steps: Step S1, pre-training the CLIP multimodal model to obtain trained weights and network structure; Step S2, constructing a task-specific image-text pair dataset; Step S3: Fine-tuning training The preloaded weights of the CLIP multimodal model are the weights of step S1. During fine-tuning training, the images in the image-text pair dataset of the specific task are input into the image encoder of the CLIP multimodal model, and the texts are input into the text encoder of the CLIP multimodal model. The image branch passes through the image encoder, fully connected layer, and normalization layer to obtain the image feature vector, which is input into the category center layer to calculate the cosine similarity between the image encoding vector and the category center vector, and obtains the cosine similarity after applying the angle boundary in the angle boundary loss calculation layer, thereby obtaining the angle boundary loss. The text branch passes through the text encoder, fully connected layer, and normalization layer to obtain the text feature vector, which is used together with the image feature vector output by the normalization layer of the image branch to calculate the contrast loss of the text-image pair; In the weighted loss layer, the contrast loss and angle boundary loss of the text-image pair are weighted to get the total loss; Step S4: Reasoning application The fine-tuned trained model weights are saved according to the standard CLIP multimodal model network structure, and then deployed to practical applications according to the standard CLIP mode.

2. The fine-tuning optimization method for improving the fine-grained discrimination capability of a multimodal model as claimed in claim 1, characterized in that: In the step S1, the publicly available pre-trained CLIP multimodal model weights and network structure are used, or training is performed on a large-scale image-text pair dataset to obtain the pre-trained CLIP multimodal model weights and network structure.

3. The fine-tuning optimization method for improving the fine-grained discrimination capability of a multimodal model as claimed in claim 1, characterized in that: The image encoder is built on top of VisionTransformer to extract feature representations of images.

4. The fine-tuning optimization method for improving the fine-grained discrimination capability of a multimodal model as claimed in claim 1, characterized in that: The text encoder is built on top of the Transformer and is used to extract feature representations of text.

5. The fine-tuning optimization method for improving the fine-grained discrimination capability of a multimodal model according to any one of claims 1 to 4, characterized in that: In step S2, a task-specific image-text pair dataset is collected and constructed, where each pair of data contains an image and a series of related text labels. i ,text i }, i = 1 ~ M, M is the total number of sample pairs. Assuming there are C categories, the dataset should cover a variety of categories and scenarios to ensure that the model can learn generalized feature representations.

6. The fine-tuning optimization method for improving the fine-grained discrimination capability of a multimodal model as claimed in claim 5, characterized in that: In step S3, a new loss function is constructed by combining contrastive learning loss and angle boundary loss for training image encoder and text encoder; based on the pre-trained model, the selected data set and loss function are used for fine-tuning to adjust the model parameters to adapt to the specific task; the new loss function is as follows: L total =(1-λ)·L CLIP +λ·L Arc λ is a hyperparameter that controls the impact of Arc loss on the total loss.

7. The fine-tuning optimization method for improving the fine-grained discrimination capability of a multimodal model as claimed in claim 6, characterized in that: λ is set to 0.

4.

8. The fine-tuning optimization method for improving the fine-grained discrimination capability of a multimodal model as claimed in claim 6, characterized in that: The contrastive learning loss for text-image pairs is calculated as follows: in, v i and t i They are the outputs of images and texts after passing through their respective encoders; and They are v i and t i The normalized result; sim(a,b) is the similarity between vectors; τ is a temperature hyperparameter that controls the magnitude of the discriminability; N is the batch size.

9. The fine-tuning optimization method for improving the fine-grained discrimination capability of a multimodal model as claimed in claim 6, characterized in that: In the image branch, an auxiliary angle boundary loss L based on angle boundary is added Arc , first calculate the cosine similarity between the normalized image vector and the category center, the category center vector is d is the length of the normalized vector, C is the number of categories, and each column is a category center vector w j , the value range of j is an integer from 1 to C, corresponding to the number of categories; the angle boundary loss L Arc The calculation steps are as follows: S31, Normalization: Image encoding vector v i and the category center vector w j All are normalized: S32, cosine similarity: Calculate the normalized cosine similarity: S33, cosine similarity after applying angle boundary: Apply an angular margin m to the similarity of the correct category: where y i Yes i The corresponding true category number, y i The value range of is an integer between 1 and C; S34, the final angle boundary loss calculation is as follows: Where N is the batch size and s is a hyperparameter, which is a scaling factor used to amplify feature normalization.

10. The fine-tuning optimization method for improving the fine-grained discrimination capability of a multimodal model as claimed in claim 9, characterized in that: The practical application of step S4 includes: image retrieval system and automatic annotation tool.

Images