Prompt learning method based on adaptive visual residual aggregation network

Abstract

The invention discloses a prompt learning method based on an adaptive visual residual aggregation network, and belongs to the technical field of deep learning. According to the method, a visual residual aggregation module and a semantic consistency module are introduced into a CLIP model, classification detection is carried out on new classes according to problems existing in a large model actual application scene, firstly, the residual aggregation module can be used for achieving self-adaptive fusion of class irrelevant features, and therefore generalization knowledge is effectively reserved; then, a learnable linear layer is introduced through a semantic consistency module, the adaptive features obtained by the residual aggregation module are input into the semantic consistency module, and semantic consistency loss and real labels are added to train generalization text prompts; and finally, through comparative learning, a final classification result is obtained. According to the method provided by the invention, when a classification task is carried out, especially new class detection is carried out, text prompts can be trained by effectively utilizing generalization features and semantic features forgotten by the model, and dynamic balance between generalization ability and semantic recognition ability of the model is realized, so that the model is more adaptive to downstream tasks, and the classification efficiency is improved. And the accuracy of the model in novelty is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning, and specifically designs a prompt learning method based on an adaptive visual residual aggregation network. Background Art

[0002] At present, through the training of a large number of image-text pairs, the visual-language model has stored a large amount of general knowledge and has a strong generalization ability for downstream tasks. However, a large amount of redundant knowledge can easily have a negative effect on the recognition accuracy of downstream tasks. The commonly used method is to use the prompt learning method to train the model to make it tend to be conducive to the distribution of downstream task features.

[0003] Hint learning is a fast and effective method. It does not require fine-tuning the entire network from scratch. It only needs to train the corresponding text hints or visual hints. And it requires less data, which greatly reduces the high data annotation cost. The existing hint learning methods mainly transfer knowledge between visible and invisible classes by training a general variable text on a small number of visible classes. However, from the perspective of training source data, current methods all use deep features for training. As the network deepens, the receptive field becomes larger, and the feature map obtained by the deep network has richer global information, which can capture more semantic information of class correlation. Therefore, deep features naturally have strong class correlation, and the trained model will be more biased towards the distribution of related classes. This will lead to classification bias in the model, low accuracy on invisible classes, and it is difficult to achieve a generalization balance between visible and invisible classes, especially the recognition performance on invisible classes drops sharply. The shallow network has a smaller receptive field and a smaller overlapping area of ​​the receptive field, which can utilize more fine-grained feature information. It includes the most basic generalization features of the image, such as texture / color / background. It has a higher resolution, contains more position and detail information, and has lower semantics. It can effectively transfer generalization information between visible and invisible classes. However, as the network goes deeper, it is difficult to retain shallow generalization features.

[0004] Based on this, the present invention designs a prompt learning method based on an adaptive visual residual aggregation network, and introduces class-independent features in the final feature layer to train a universal text prompt, which maintains significant semantic features while also retaining general knowledge, thereby optimizing the recognition effect of downstream tasks, especially the recognition of invisible classes. Summary of the invention

[0005] In order to improve the generalization of learnable text prompts, especially to optimize the accuracy on invisible classes in downstream tasks, the present invention proposes a prompt learning method based on an adaptive visual residual aggregation network. The method introduces two modules, namely residual and semantic consistency modules. The residual generalization module trains text prompts by adaptively mapping shallow features, that is, class-irrelevant features, to deep features to increase its generalization performance on invisible classes. Since the pre-trained network is trained based on a large number of image-text pairs, the recognition accuracy is strongly correlated with the matching degree of semantic and image features. In order to optimize semantic deviations and maintain a dynamic balance between generalized knowledge and semantic features, a semantic consistency module is introduced. Finally, semantic consistency loss and contrastive learning loss are used to optimize text prompts and learnable parameters, so that the general model is more suitable for downstream tasks. The technical problem proposed by the present invention is solved as follows:

[0006] Specific content:

[0007] A cue learning method based on an adaptive visual residual aggregation network comprises the following steps:

[0008] Step 1: Obtain the required classification data set, and perform preliminary processing on the images in the data set to divide them into a base class training set, a base class test set, and a new class test set;

[0009] Step 2: preprocess the obtained classification data, convert each image into multiple image blocks, normalize them, and input them into the CLIP model;

[0010] Step 3: Use the residual aggregation module to adaptively extract shallow generalization features from the image data obtained in step 2, and fuse them with deep semantic features to effectively extract features in the image that are generalizable and can be applied to new class recognition at the same time;

[0011] Step 4: Get the residual adaptive features obtained in step 3 and the real labels in the data set and input them into the linear layer of the semantic consistency module. Then, add the learnable semantic consistency loss to the overall contrastive learning loss to improve the model's ability to extract semantic information, perform feature-level constraints, and extract features with strong self-distinguishing commonalities;

[0012] Step 5: Input the features obtained in step 4 into the classifier network to obtain the final classification result. The loss functions used include the classification loss of contrastive learning, text generalization loss, and semantic consistency loss.

[0013] Step 6: Train the adaptive visual residual aggregation network constructed in steps 3, 4, and 5 on the base class training set.

[0014] Step 1 of the present invention comprises:

[0015] Step 1-1, obtain downstream task datasets, including ImageNet, Caltech, OxfordPets, StanfordCars, Flowers, Food101, FGVCAircraft, EuroSAT, UCF101, DTD and SUN397, and perform preliminary processing on the datasets;

[0016] Step 1-2: After step 1-1, the data set obtained is divided into a base class training set, a base class test set and a new class test set to facilitate subsequent training and testing of the network.

[0017] Step 2 of the present invention comprises:

[0018] Step 2-1, use imageio and opencv to read the image, and read the corresponding image file according to the input file name and file path;

[0019] Step 2-2, before the image is divided into blocks, the input image is usually normalized. The input image is first divided into non-overlapping image blocks of fixed size, each image block represents the pixel value of a local area, and then the data is read according to the file corresponding to the index;

[0020] In step 2-3, the 16x16 image block is flattened into a 256-dimensional vector and mapped to the specified high-dimensional space through a linear transformation. Then, a learnable or fixed position information encoding is added to each image block to ensure that the position information of the image block can be captured by the model.

[0021] Step 3 of the present invention comprises:

[0022] Step 3-1, connect the shallow features of the CLIP model with the features of the classification layer through a branch structure similar to the residual network, and perform adaptive fusion of features. The calculation formula is as follows:

[0023] fea (fin) =Fa,

[0024] F=[fea 1 ,fea 2 ,…,fea n ,fea (last) ]

[0025]

[0026] Where n is the number of shallow layers; fea 1 to fea n It is a shallow feature; (last) is the feature of the last layer; a 1 to a nis an adaptive parameter, which is trained together with the network to balance the proportion of generalization features and class-related features. m The initial experience value of fea is set to 0.1; (fin) is the feature after adaptive fusion.

[0027] Step 3-2, then, replace the deep features with the fused features for contrastive learning and input into the semantic consistency module.

[0028] Step 4 of the present invention comprises:

[0029] Step 4-1: For the features obtained in step 3, input it and the true label into the learnable linear layer of the semantic consistency module for processing.

[0030] Step 4-2, then, the generation of image feature space and the training of learnable cues are carried out simultaneously under the guidance of the linear classification layer. The training goal is to minimize the semantic consistency loss to maintain high semantic discriminability. The formula for calculating semantic consistency loss is as follows:

[0031]

[0032] Among them, N m is the total number of samples; is the true label of the i-th sample; The i-th sample is labeled as c k probability.

[0033] Step 5 of the present invention comprises:

[0034] Step 5-1, train the adaptive parameters and text prompts together with the results of step 4 and the global loss, and freeze other model parameters. The final loss calculation formula is as follows:

[0035] L res =L ce +λL kg +βL sem

[0036] Among them, L res is the final optimization target of the model; λ and β represent weights, which are used to balance the generalization space and the semantic space. This multimodal loss maintains the generalization and uniqueness of features in the text and image ends, respectively, making the trained model more suitable for the distribution of new categories. In addition, L kg is the text generalization loss and L ce is the standard cross entropy loss for contrastive learning, and is calculated as follows:

[0037]

[0038] in, represents the learnable textual hint embedding of the i-th image obtained by the text encoder, Represents a fixed text embedding obtained from a text encoder:

[0039]

[0040] Among them, x kgcoop ∈X, X is the image dataset, and τ is the training temperature.

[0041] The present invention proposes a cue learning method based on an adaptive visual residual aggregation network, which has the following advantages and beneficial effects compared with the prior art:

[0042] The present invention proposes a cue learning method for an adaptive visual residual aggregation network, which can effectively adapt to recognition tasks of unseen classes. Compared with existing methods, the method proposed in the present invention can effectively utilize the general knowledge of the model while correcting the semantic deviation problem when performing unseen class recognition, so that the learned cue can maintain important semantic features, improve the accuracy, and effectively guide the pre-trained visual language model to adapt to downstream recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is the overall flow diagram of the method of the present invention

[0044] Figure 2 is a schematic diagram of the residual adaptive module;

[0045] Figure 3 This is a diagram of the semantic adaptation module.

[0046] Figure 4 This is a schematic diagram of the attention mechanism results DETAILED DESCRIPTION

[0047] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0048] The present invention is implemented including:

[0049] The overall process of the method proposed by the present invention is as follows Figure 1 As shown, the specific construction steps of forest fire detection in the embodiment of the present invention are as follows:

[0050] Step 1: Obtain the required classification data set, and perform preliminary processing on the videos in the data set, dividing them into base class training set, base class test set and new class test set, so as to facilitate the subsequent training and testing of the network;

[0051] Step 2: preprocess the data obtained in step 1, convert each image into multiple image blocks, normalize them, and input them into the CLIP model;

[0052] Step 3: Use the residual adaptive module to extract features from the image data obtained in step 2, and connect the shallow features of the CLIP model with the features of the classification layer through a branch structure similar to the residual network to perform adaptive fusion of features. Then replace the deep features with the fused features for comparative learning and input into the semantic consistency module. The schematic diagram of the residual adaptive module is shown in the figure. Figure 2 As shown;

[0053] Step 4: Input the features obtained in step 3 and the true label into the learnable linear layer of the semantic consistency module for processing. Then, the generation of image feature space and the training of learnable hints are carried out simultaneously under the guidance of the linear classification layer. Figure 3 As shown;

[0054] Step 5: Input the features obtained in step 4 into the classifier network to obtain the final classification result. The loss functions used include the classification loss of contrastive learning, text generalization loss, and semantic consistency loss.

[0055] Step 6: Train the adaptive visual residual aggregation network built in steps 3, 4, and 5 on the base class training set. After training, the attention is more focused on the classified objects. The attention diagram on the network layer is as follows: Figure 4 shown.

[0056] Step 1 of the present invention comprises:

[0057] Step 1-1, obtain downstream task datasets, including ImageNet, Caltech, OxfordPets, StanfordCars, Flowers, Food101, FGVCAircraft, EuroSAT, UCF101, DTD and SUN397, and perform preliminary processing on the datasets;

[0058] Step 1-2: After step 1-1, the data set obtained is divided into a base class training set, a base class test set and a new class test set to facilitate subsequent training and testing of the network.

[0059] Step 2 of the present invention comprises:

[0060] Step 2-1, use imageio and opencv to read the image, and read the corresponding image file according to the input file name and file path;

[0061] Step 2-2, before the image is divided into blocks, the input image is usually normalized. The input image is first divided into non-overlapping image blocks of fixed size, each image block represents the pixel value of a local area, and then the data is read according to the file corresponding to the index;

[0062] In step 2-3, the 16x16 image block is flattened into a 256-dimensional vector and mapped to the specified high-dimensional space through a linear transformation. Then, a learnable or fixed position information encoding is added to each image block to ensure that the position information of the image block can be captured by the model.

[0063] Step 3 of the present invention comprises:

[0064] Step 3-1, connect the shallow features of the CLIP model with the features of the classification layer through a branch structure similar to the residual network, and perform adaptive fusion of features. The calculation formula is as follows:

[0065] fea (fin) =Fa,

[0066] F=[fea 1 ,fea 2 ,…,fea n ,fea (last) ]

[0067]

[0068] Where n is the number of shallow layers; fea 1 to fea n It is a shallow feature; (last) is the feature of the last layer; a 1 to a n is an adaptive parameter, which is trained together with the network to balance the proportion of generalization features and class-related features. m The initial experience value of fea is set to 0.1; (fin) is the feature after adaptive fusion.

[0069] Step 3--2, then, replace the deep features with the fused features for contrastive learning and input into the semantic consistency module.

[0070] Step 4 of the present invention comprises:

[0071] Step 4-1: For the features obtained in step 3, input it and the true label into the learnable linear layer of the semantic consistency module for processing.

[0072] Step 4-2, then, the generation of image feature space and the training of learnable cues are carried out simultaneously under the guidance of the linear classification layer. The training goal is to minimize the semantic consistency loss to maintain high semantic discriminability. The formula for calculating semantic consistency loss is as follows:

[0073]

[0074] Among them, N m is the total number of samples; is the true label of the i-th sample; The i-th sample is labeled as c k probability.

[0075] Step 5 of the present invention comprises:

[0076] Step 5-1, train the adaptive parameters and text prompts together with the results of step 4 and the global loss, and freeze other model parameters. The final loss calculation formula is as follows:

[0077] L res =L ce +λL kg +βL sem

[0078] Among them, L res is the final optimization target of the model; λ and β represent weights, which are used to balance the generalization space and the semantic space. This multimodal loss maintains the generalization and uniqueness of features in the text and image ends, respectively, making the trained model more suitable for the distribution of new categories. In addition, L kg is the text generalization loss and L ce is the standard cross entropy loss for contrastive learning, and is calculated as follows:

[0079]

[0080] in, represents the learnable textual hint embedding of the i-th image obtained by the text encoder, Represents a fixed text embedding obtained from a text encoder:

[0081]

[0082] Among them, x kgcoop ∈X, X is the image dataset, and τ is the training temperature.

[0083] The present invention proposes a cue learning method for an adaptive visual residual aggregation network, which can effectively adapt to recognition tasks of unseen classes. Compared with existing methods, the method proposed in the present invention can effectively utilize the general knowledge of the model while correcting the semantic deviation problem when performing unseen class recognition, so that the learned cue can maintain important semantic features, improve the accuracy, and effectively guide the pre-trained visual language model to adapt to downstream recognition tasks.

[0084] The present invention provides a prompt learning method for an adaptive visual residual aggregation network. The above description is only an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A cue learning method based on an adaptive visual residual aggregation network, comprising the following steps: Step 1: Obtain the required classification data set, and perform preliminary processing on the images in the data set to divide them into a base class training set, a base class test set, and a new class test set; Step 2: preprocess the obtained classification data, convert each image into multiple image blocks, normalize them, and input them into the CLIP model; Step 3: Use the residual aggregation module to adaptively extract shallow generalization features from the image data obtained in step 2, and fuse them with deep semantic features to effectively extract features in the image that are generalizable and can be applied to new class recognition at the same time; Step 4: Get the residual adaptive features obtained in step 3 and the real labels in the data set and input them into the linear layer of the semantic consistency module. Then, add the learnable semantic consistency loss to the overall contrastive learning loss to improve the model's ability to extract semantic information, perform feature-level constraints, and extract features with strong self-distinguishing commonalities; Step 5: Input the features obtained in step 4 into the classifier network to obtain the final classification result. The loss functions used include the classification loss of contrastive learning, text generalization loss, and semantic consistency loss. Step 6: Train the adaptive visual residual aggregation network constructed in steps 3, 4, and 5 on the base class training set.

2. The method according to claim 1, characterized in that Step 2 includes: Step 2-1, use imageio and opencv to read the image, and read the corresponding image file according to the input file name and file path; Step 2-2, before the image is divided into blocks, the input image is usually normalized. The input image is first divided into non-overlapping image blocks of fixed size, each image block represents the pixel value of a local area, and then the data is read according to the file corresponding to the index; In step 2-3, the 16x16 image block is flattened into a 256-dimensional vector and mapped to the specified high-dimensional space through a linear transformation. Then, a learnable or fixed position information encoding is added to each image block to ensure that the position information of the image block can be captured by the model.

3. The method according to claim 1, characterized in that Step 3 includes: Step 3-1, connect the shallow features of the CLIP model with the features of the classification layer through a branch structure similar to the residual network, and perform adaptive fusion of features. The calculation formula is as follows: fea (fin) =Do, F=[fea1,fea2,…,fea n ,ugly (last) ] Among them, n is the number of shallow layers; fea1 to fea n It is a shallow feature; (last) is the feature of the last layer; a1 to a n is an adaptive parameter, which is trained together with the network to balance the proportion of generalization features and class-related features. m The initial experience value of fea is set to 0.1; (fin ) is the feature after adaptive fusion. Step 3-2, then, replace the deep features with the fused features for contrastive learning and input into the semantic consistency module.

4. The method according to claim 1, characterized in that Step 4 includes: Step 4-1: For the features obtained in step 3, input it and the true label into the learnable linear layer of the semantic consistency module for processing. Step 4-2, then, the generation of image feature space and the training of learnable cues are carried out simultaneously under the guidance of the linear classification layer. The training goal is to minimize the semantic consistency loss to maintain high semantic discriminability. The formula for calculating semantic consistency loss is as follows: Among them, N m is the total number of samples; is the true label of the i-th sample; The i-th sample is labeled as c k probability.

5. The method according to claim 1, characterized in that Step 5 includes: Step 5-1, train the adaptive parameters and text prompts together with the results of step 4 and the global loss, and freeze other model parameters. The final loss calculation formula is as follows: L res =L ce +λL kg +βL sem Among them, L res is the final optimization target of the model; λ and β represent weights, which are used to balance the generalization space and the semantic space. This multimodal loss maintains the generalization and uniqueness of features in the text and image ends, respectively, making the trained model more suitable for the distribution of new categories. In addition, L kg is the text generalization loss and L ce is the standard cross entropy loss for contrastive learning, and is calculated as follows: in, represents the learnable textual hint embedding of the i-th image obtained by the text encoder, Represents a fixed text embedding obtained from a text encoder: Among them, x kgcoop ∈X, X is the image dataset, and τ is the training temperature.

Images