A hint learning method based on an adaptive visual residual aggregation network

Abstract

The application discloses a prompt learning method based on an adaptive visual residual aggregation network and belongs to the technical field of deep learning.The application introduces a visual residual aggregation module and a semantic consistency module into a CLIP model to classify and detect new classes according to problems existing in actual application scenarios of a large model.First, the residual aggregation module can be used to realize adaptive fusion of class-independent features, thereby effectively retaining generalization knowledge.Then, a learnable linear layer is introduced into the semantic consistency module through the semantic consistency module, adaptive features obtained by the residual aggregation module are input into the semantic consistency module, semantic consistency loss and real labels are added, and generalization text prompts are trained.Finally, through contrastive learning, the final classification result is obtained.The method proposed in the application can effectively utilize generalization features and semantic features forgotten by the model to train text prompts when performing a classification task, especially for new class detection, realizes dynamic balance between generalization ability and semantic recognition ability of the model, makes the model more adaptive to downstream tasks, and effectively improves the accuracy of the model on new types.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of deep learning, and specifically designs a prompt learning method based on an adaptive visual residual aggregation network. BACKGROUND

[0002] At present, through a large number of image text pairs training, the visual-linguistic model has stored a large amount of general knowledge, and has a strong generalization ability for downstream tasks. However, a large amount of redundant knowledge is easy to have a negative effect on the recognition accuracy of the downstream task. The commonly used method is to use the prompt learning method to train the model to tend to have a beneficial direction to the feature distribution of the downstream task.

[0003] Prompt learning is a fast and effective method, which does not need to start fine-tuning the entire network, but only needs to train the corresponding text prompt or visual prompt. And the required data amount is less, which greatly reduces the high data annotation cost. The existing prompt learning method mainly trains a general variable text on a small number of visible classes, and transfers knowledge in the visible class and the invisible class. However, from the training source data, the current method is to train the deep features. With the deepening of the network, the receptive field becomes larger, and the global information of the feature map obtained by the deep network is more rich, which can capture more class-related semantic information. Therefore, the deep features naturally have strong class correlation, and the trained model will be more biased to the distribution of the related class. This will cause the model to have classification bias, and the accuracy on the invisible class is low, which is difficult to achieve the generalization balance of the visible class and the invisible class, especially the recognition performance on the invisible class drops sharply. The receptive field of the shallow network is smaller, and the overlapping area of the receptive field is also smaller, which can utilize more fine-grained feature information. It includes the most basic texture / color / background and other generalization features of the image, its resolution is higher, and it contains more position and detail information, and its semanticity is lower, which can effectively transfer the generalization information between the visible class and the invisible class. However, with the deepening of the network, it is difficult to retain the generalization features of the shallow layer.

[0004] Based on this, the application designs a prompt learning method based on an adaptive visual residual aggregation network, introduces class-unrelated features in the last feature layer, trains a general text prompt, retains significant semantic features while retaining generalization general knowledge, thereby optimizing the recognition effect of the downstream task, especially the recognition of the invisible class. SUMMARY

[0005] In order to improve the generalization of learnable text prompts, especially to optimize the accuracy on unseen classes in downstream tasks, the application proposes a prompt learning method based on an adaptive visual residual aggregation network. The method introduces two modules, residual and semantic consistency modules. The residual generalization module trains the text prompt by adaptively mapping and fusing the shallow features, i.e. class-irrelevant features, to the deep features, thereby increasing the generalization performance on unseen classes. Since the pre-trained network is trained based on a large number of image-text pairs, the recognition accuracy has a strong correlation with the matching degree of semantic and image features. In order to optimize the semantic deviation and maintain the dynamic balance between the generalization knowledge and the semantic features, the semantic consistency module is introduced. Finally, the semantic consistency loss and the contrastive learning loss are used to optimize the text prompt and the learnable parameters, so that the general model is more suitable for downstream tasks. The technical problems of the application are solved as follows:

[0006] Specific content:

[0007] A prompt learning method based on an adaptive visual residual aggregation network, comprising the following steps:

[0008] Step 1, obtaining the required classification data set, and preliminarily processing the pictures in the data set, dividing the base class training set, the base class test set and the new class test set;

[0009] Step 2, pre-processing the obtained classification data, converting each picture into multiple image blocks, and performing normalization processing, and inputting the CLIP model;

[0010] Step 3, using the residual aggregation module to adaptively extract the shallow generalization features of the image data obtained in step 2, and fusing the deep semantic features, effectively extracting the features in the image which have generalization and can be applied to new class recognition at the same time;

[0011] Step 4, obtaining the residual adaptive features obtained in step 3 and the real labels in the data set together input into the linear layer of the semantic consistency module. Then, the learnable semantic consistency loss is added to the overall contrastive learning loss, the model's ability to extract semantic information is improved, and the features with strong self-differentiation commonality are extracted through feature-level constraint;

[0012] Step 5, inputting the features obtained in step 4 into the classifier network to obtain the final classification result, and the loss function adopted includes the classification loss of contrastive learning, the text generalization loss and the semantic consistency loss;

[0013] Step 6, training the adaptive visual residual aggregation network built in steps 3, 4 and 5 on the base class training set.

[0014] The step 1 of the application comprises:

[0015] Step 1-1, obtain a downstream task dataset, including ImageNet, Caltech, OxfordPets, StanfordCars, Flowers, Food101, FGVCAircraft, EuroSAT, UCF101, DTD and SUN397, and perform preliminary processing on the dataset;

[0016] Step 1-2, the dataset obtained after step 1-1 is divided into base class training set, base class test set and new class test set, which facilitates the training and testing of the network later.

[0017] The step 2 of the application comprises:

[0018] Step 2-1, imageio and opencv are used for image reading, and the corresponding picture file is read according to the input file name and file path;

[0019] Step 2-2, before image blocking, the input image usually undergoes pixel value normalization. The input image is first divided into fixed size non-overlapping image blocks, and each image block represents the pixel value of a local area, and then the data reading is completed according to the index corresponding file;

[0020] Step 2-3, the 16x16 image block is flattened into a 256-dimensional vector, and is mapped to a specified high-dimensional space through a linear transformation. Then, the learnable or fixed position information coding is added to each image block, so that the position information of the image block can be captured by the model.

[0021] The step 3 of the application comprises:

[0022] Step 3-1, the shallow features of the CLIP model are connected with the features of the classification layer through a branch structure similar to the residual network, and the adaptive fusion of the features is performed, and the calculation formula is as follows:

[0023] fea (fin) = Fa,

[0024] F=[fea1,fea2,…,fea n ,fea (last) ]

[0025]

[0026] Wherein, n is the number of shallow layers; fea1 to fea n are shallow features; fea (last) is the feature of the last layer; a1 to a n are adaptive parameters, which are trained with the network and used to balance the proportion of generalization features and class-related features, and here a mThe initial experience value of the image feature is set to 0.1; fea (fin) The adaptive fused feature.

[0027] Step 3-2, then, replace the deep feature with the fused feature for contrastive learning and input semantic consistency module.

[0028] The step 4 of the application comprises:

[0029] Step 4-1, the feature obtained by step 3 is input into the learnable linear layer of the semantic consistency module for processing.

[0030] Step 4-2, then, the generation of the image feature space and the training of the learnable prompt are carried out simultaneously under the guidance of the linear classification layer. The training purpose is to minimize the semantic consistency loss to maintain high semantic distinguishability. The formula of the semantic consistency loss is as follows:

[0031]

[0032] Where, N m is the total number of samples; is the true label of the i-th sample; is the probability that the i-th sample is labeled as c k .

[0033] The step 5 of the application comprises:

[0034] Step 5-1, the results of step 4 are trained together with the global loss to adapt the parameters and text prompts, and other model parameters are frozen. The final loss calculation formula is as follows:

[0035] L res = L ce + λL kg + βL sem

[0036] Where, L res is the final optimization target of the model; λ and β represent the weights for balancing the generalization space and the semantic space. This multi-modal loss respectively maintains the generality and uniqueness of the features at the text and image ends, making the trained model more suitable for the distribution of new categories. In addition, L kg is the text generalization loss and L ce is the standard cross-entropy loss of contrastive learning, and the formula is as follows:

[0037]

[0038] Where, represents the i-th image learnable text prompt embedding obtained by the text encoder, representing a fixed text embedding obtained from a text encoder:

[0039]

[0040] where x kgcoop ∈X, X is an image dataset, and τ is a training temperature.

[0041] The application proposes a prompt learning method based on an adaptive visual residual aggregation network, which has the following advantages and beneficial effects compared with the prior art:

[0042] The application proposes a prompt learning method based on an adaptive visual residual aggregation network, which can effectively adapt to the recognition task of unseen classes. Compared with existing methods, the method proposed by the application can effectively utilize the general knowledge of the model while correcting the semantic bias problem when performing unseen class recognition, so that the learned prompt retains important semantic features, improves the accuracy, and effectively guides the pre-trained visual language model to adapt to the downstream recognition task. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is a schematic diagram of the overall process of the method of the application

[0044] Figure 2 is a schematic diagram of the residual adaptive module

[0045] Figure 3 is a schematic diagram of the semantic adaptive module

[0046] Figure 4 is a schematic diagram of the attention mechanism result DETAILED DESCRIPTION

[0047] The preferred embodiments of the application will be described in detail below with reference to the accompanying drawings, so that the advantages and features of the application can be more easily understood by those skilled in the art, and the scope of protection of the application can be more clearly defined.

[0048] The application implementation includes:

[0049] The overall process of the method of the application is shown in Figure 1 The specific construction steps of the forest fire detection in the embodiment of the application are as follows:

[0050] Step 1, obtain the required classification dataset, and preliminarily process the videos in the dataset, divide the base class training set, the base class test set and the new class test set, which is convenient for training and testing the network later;

[0051] Step 2, pre-process the data obtained in step 1, convert each picture into multiple image blocks, and perform normalization processing, and input the CLIP model;

[0052] Step 3, the residual adaptive module is used to extract features from the image data obtained in step 2, the shallow features of the CLIP model are connected to the features of the classification layer through a branch structure similar to the residual network, and the adaptive fusion of the features is performed. Then the fused features are used to replace the deep features for contrastive learning and input semantic consistency module. The schematic diagram of the residual adaptive module is shown in Figure 2 ;

[0053] Step 4, the features obtained by step 3 are input into the learnable linear layer of the semantic consistency module for processing. Then, the generation of the image feature space and the training of the learnable prompt are carried out online under the guidance of the linear classification layer. The schematic diagram of the structure of the semantic consistency module is shown in Figure 3 ;

[0054] Step 5, the features obtained in step 4 are input into the classifier network to obtain the final classification result, and the loss function includes the classification loss of contrastive learning, the text generalization loss and the semantic consistency loss;

[0055] Step 6, the adaptive visual residual aggregation network built in steps 3, 4 and 5 is trained on the base class training set, and the attention of the trained network layer is more concentrated. The schematic diagram of the attention of the network layer is shown in Figure 4 .

[0056] The step 1 of the application comprises:

[0057] Step 1-1, obtain the downstream task data set, including ImageNet, Caltech, OxfordPets, StanfordCars, Flowers, Food101, FGVCAircraft, EuroSAT, UCF101, DTD and SUN397, and preliminarily process the data set;

[0058] Step 1-2, the data set obtained after step 1-1 is divided into base class training set, base class test set and new class test set, which facilitates the training and testing of the network later.

[0059] The step 2 of the application comprises:

[0060] Step 2-1, use imageio and opencv to read images, and read the corresponding picture file according to the input file name and file path;

[0061] Step 2-2, before image blocking, the input image usually undergoes pixel value normalization. The input image is first divided into fixed-size non-overlapping image blocks, and each image block represents the pixel value of a local area, and then the data reading is completed according to the index corresponding file.

[0062] Step 2-3, flatten the 16x16 image block into a 256-dimensional vector and map it to a designated high-dimensional space through a linear transformation. Then add a learnable or fixed position information code to each image block to ensure that the position information of the image block can be captured by the model.

[0063] The step 3 of the present application comprises:

[0064] Step 3-1, connect the shallow features of the CLIP model and the features of the classification layer through a branch structure similar to the residual network, and perform adaptive fusion of the features, the calculation formula is as follows:

[0065] fea (fin) =Fa,

[0066] F=[fea1,fea2,…,fea n ,fea (last) ]

[0067]

[0068] Wherein, n is the number of shallow layers; fea1 to fea n are shallow features; fea (last) is the feature of the last layer; a1 to a n are adaptive parameters, which are trained with the network to balance the proportion of generalization features and class-related features, and the initial experience value of a m is set to 0.1; fea (fin) is the feature after adaptive fusion.

[0069] Step 3-2, then, replace the deep features with the fused features for contrastive learning and input semantic consistency module.

[0070] The step 4 of the present application comprises:

[0071] Step 4-1, input the feature obtained through step 3 and the real label into the learnable linear layer of the semantic consistency module for processing.

[0072] Step 4-2, then, the generation of image feature space and the training of learnable prompt are carried out simultaneously under the guidance of the linear classification layer. The training purpose is to minimize the semantic consistency loss to maintain high semantic distinguishability. The calculation formula of semantic consistency loss is as follows:

[0073]

[0074] Wherein, N m is the total number of samples; is the real label of the i-th sample; is the probability that the i-th sample is labeled as c k .

[0075] The step 5 of the present application comprises:

[0076] Step 5-1, the results of step 4 and the global loss are used to train adaptive parameters and text prompts, and other model parameters are frozen. The final loss calculation formula is as follows:

[0077] L res = L ce + λL kg + βL sem

[0078] Wherein, L res is the final optimization target of the model; λ and β represent the weight, used to balance the generalization space and the semantic space. This multi-modal loss respectively maintains the generality and uniqueness of the features at the text and image ends, so that the trained model is more suitable for the distribution of new categories. In addition, L kg is the text generalization loss and L ce is the standard cross-entropy loss of contrast learning, and the calculation formula is as follows:

[0079]

[0080] Wherein, represents the i-th image of the learnable text prompt embedding obtained by the text encoder, represents the fixed text embedding obtained from the text encoder:

[0081]

[0082] Wherein, x kgcoop ∈X, X is an image data set, and τ is a training temperature.

[0083] The present application proposes a prompt learning method of an adaptive visual residual aggregation network, which can effectively adapt to the recognition task of unseen classes. Compared with the existing method, the method proposed by the present application can effectively utilize the general knowledge of the model while correcting the semantic deviation problem when performing unseen class recognition, so as to keep the learned prompt important semantic features, improve the accuracy, and effectively guide the pre-trained visual language model to adapt to the downstream recognition task.

[0084] The application provides a hint learning method of an adaptive visual residual aggregation network, and the above description is only an embodiment of the application, and does not limit the patent range of the application, and any equivalent structure or equivalent process transformation using the content of the specification and drawings of the application, or direct or indirect application in other related technical fields, are also included in the patent protection range of the application.

Claims

1. A prompt learning method based on an adaptive visual residual aggregation network, comprising the following steps: Step 1, obtaining the required classification data set, and preliminarily processing the pictures in the data set, dividing the base class training set, the base class test set and the new class test set; Step 2, pre-processing the obtained classification data, converting each picture into multiple image blocks, and performing normalization processing, and inputting the CLIP model; Step 3, using the residual aggregation module to adaptively extract the shallow generalization features of the image data obtained in step 2, and fuse with the deep semantic features, effectively extracting the features in the image which have generalization and can be applied to new class recognition at the same time; through the branch structure similar to the residual network, the shallow features of the CLIP model are connected with the features of the classification layer, and the adaptive fusion of the features is carried out, and the calculation formula is as follows: fea (fin) = Fa, F = [fea1, fea2,..., fean] (1) n , fea (last) ] Where n is the number of shallow layers; from fea1 to fea n This is a shallow feature; fea (last) Features of the last layer; a1 to a n These are adaptive parameters, trained along with the network, used to balance the ratio of generalization features to class-related features. Here, a... m The initial empirical value is set to 0.1; fea (fin) The deep features are then adapted and fused; then, the fused features are used to replace the deep features for contrastive learning and input semantic consistency module. Step 4, obtaining the residual adaptive features obtained in step 3 and the real label in the data set together input into the linear layer of the semantic consistency module; then, the learnable semantic consistency loss is added to the overall contrast learning loss, the model's ability to extract semantic information is improved, and the features with strong self-differentiation commonality are extracted through the constraint at the feature level; Step 5, inputting the features obtained in step 4 into the classifier network to obtain the final classification result, and the loss function adopted includes the classification loss of contrast learning, the text generalization loss and the semantic consistency loss; Step 6, training the adaptive visual residual aggregation network built in steps 3, 4 and 5 on the base class training set.

2. The method of claim 1, wherein Step 2 includes: Step 2-1, using imageio and opencv to read the image, according to the input file name, file path to read the corresponding picture file; Step 2-2, before image blocking, the input image will be subjected to pixel value normalization, the input image is first divided into fixed size non-overlapping image blocks, each image block represents the pixel value of the local area, then according to the index corresponding to the file, the data reading is completed; Step 2-3, flattening the 16x16 image block into a 256-dimensional vector, and mapping it to a specified high-dimensional space through a linear transformation, then adding learnable or fixed position information coding to each image block to ensure that the position information of the image block can be captured by the model.

3. The method of claim 1, wherein Step 4 includes: Step 4-1, inputting the obtained fusion features and the real label into the learnable linear layer of the semantic consistency module for processing; Step 4-2, then, the generation of image feature space and the training of learnable prompt are carried out simultaneously under the guidance of the linear classification layer, and the purpose of the training is to minimize the semantic consistency loss, and the calculation formula of the semantic consistency loss is as follows: where N m is the total number of samples; is the true label of the i-th sample; is the probability that the i-th sample is labeled as c k .

4. The method of claim 1, wherein Step 5 includes: Step 5-1, training the adaptive parameters and the text prompt together with the global loss of step 4, and freezing other model parameters, and the final loss calculation formula is as follows: L res = L ce + λL kg + βL sem wherein, L res is the final optimization target of the model; λ and β represent the weights for balancing the generalization space and the semantic space; this multi-modal loss respectively maintains the generality and uniqueness of the features at the text and image ends, making the trained model more suitable for the distribution of new categories; in addition, L kg is the text generalization loss and L ce is the standard cross-entropy loss of contrast learning, and the calculation formula is as follows: wherein, represents a learnable text prompt embedding for the i-th image obtained by the text encoder, represents a fixed text embedding obtained from the text encoder: where x kgcoop ∈ X, X is the image dataset, and τ is the training temperature.

Images