Few-sample fine tuning method and device for pre-trained visual language model and readable medium

Abstract

The invention discloses a few-sample fine tuning method and device for a pre-trained visual language model and a readable medium, and the method comprises the steps: constructing a fine tuning data set, constructing the pre-trained visual language model, a feature alignment module and a local caching module, and enabling the pre-trained visual language model to comprise an image encoder, a text encoder and a feature aggregation module; and based on the fine tuning data set, performing fine tuning on the pre-trained visual language model through a feature alignment module and a local cache module to obtain a comparison pre-trained visual language model and a last updated storage matrix. Firstly, global features and local features are converted into a text embedding space through a feature comparison module, fine-grained semantic information is extracted from the local features of image branches through packaging of a local cache module, and then the fine-grained semantic information and corresponding text embedding are spliced. And fusing global and local information through a feature aggregation module to enhance the representation ability of the text branches.

Description

Technical Field

[0001] The present invention relates to the field of image and text processing, and in particular to a few-sample fine-tuning method, device and readable medium for a pre-trained visual language model. Background Art

[0002] For traditional computer vision systems, learning of visual representations relies entirely on a predefined set of labels, which limits their ability to scale to unseen categories. Visual language models make up for this shortcoming by introducing natural language supervision, thereby helping visual systems to master more visual concepts. These models are pre-trained in a self-supervised manner on a large amount of raw text and image pairing data obtained from the Internet, optimizing the image encoder and text encoder through contrastive loss, and aligning text and image features into a unified embedding space. Thanks to large-scale datasets and contrastive training objectives, the visual representations learned by visual language models are connected to language and have high generalization capabilities, which can be widely used in various downstream visual tasks such as object detection, semantic segmentation, and depth estimation. Despite these powerful capabilities, migrating these visual language models to downstream tasks in few-sample scenarios still faces severe challenges. Because the amount of data is greatly reduced compared to the pre-training stage, the generalization ability of visual language models on migration tasks is usually significantly reduced.

[0003] To date, many studies have been devoted to transferring visual-language models in the few-shot case, with much focus on using cue learning, which is a well-known strategy for adapting large pre-trained models because it does not require a complete retraining of the original model, but rather enables downstream transfer by introducing learnable textual or visual cues that go beyond the manually defined cues of the pre-trained model, such as “a photo of a{class}”.

[0004] As a preliminary exploration, CoOp added learnable markers to the predefined text prompts in the text branch. These markers are shared across all categories, capturing task-related knowledge from downstream datasets to align with image features. CoCoOp constrains image features in an instance-specific manner to reduce overfitting in few-sample scenarios and enhance generalization to unseen categories. PromptSRC further leverages pre-trained features to condition learned prompts to enhance generalization. Subsequent research efforts have attempted to extend prompt learning to multimodal branches, such as MaPLe, which extends prompt learning to both image and text branches, so that prompts can not only capture features in the text branch, but also enhance alignment between modalities.

[0005] Despite continuous progress, a common limitation of these works is that they tend to enhance high-level semantics in a coarse-grained global manner to achieve overall alignment between modalities. As a result, customized cues may fail to guide the model to recognize diverse visual concepts from local regions, limiting its generalization ability between categories with similar visual properties. To overcome this limitation, recent works such as ArGue and LLaMP have enriched textual cues with large language models (LLMs) to incorporate fine-grained category descriptions to improve transfer performance, such as Figure 1 Unfortunately, these methods require additional steps to screen the output of LLMs to ensure their relevance to the downstream dataset, thus introducing significant inference overhead. Summary of the invention

[0006] The purpose of this application is to propose a few-sample fine-tuning method, device and readable medium for a pre-trained visual language model in response to the above-mentioned technical problems.

[0007] In a first aspect, the present invention provides a few-sample fine-tuning method for a pre-trained visual language model, comprising the following steps:

[0008] Get several images, texts and the true labels corresponding to each image and build a fine-tuning dataset;

[0009] Construct a pre-trained visual language model, a feature alignment module, and a local cache module. The pre-trained visual language model includes an image encoder, a text encoder, and a feature aggregation module. The image encoder and the text encoder are respectively connected to the feature aggregation module.

[0010] Each image and text in the fine-tuning dataset is input into the pre-trained visual language model, each image is divided into several non-overlapping image blocks through the image encoder, the global features of the image and the local features of each image block are extracted respectively to form a feature set of each image, and the feature set of each image is input into the feature alignment module to obtain the converted feature set of each image; a storage matrix is ​​constructed in the local cache module, the storage matrix is ​​randomly initialized to obtain the initial storage matrix, the converted feature set of each image is traversed, the initial storage matrix is ​​updated according to the converted feature set of the first image, and the storage matrix after the last update is updated according to the converted feature set of the current image to obtain the storage matrix after the current update; each text is input into the text encoder to obtain the text embedding, the storage matrix after the current update and the text embedding are input into the feature aggregation module to obtain the current aggregated features, the predicted label corresponding to the current image is calculated according to the current aggregated features and the global features of the current image, the loss function is constructed using the predicted label and the true label, and the pre-trained visual language model is fine-tuned based on the loss function to obtain the comparison between the pre-trained visual language model and the storage matrix after the last update.

[0011] Preferably, the feature alignment module is a multilayer perceptron with two hidden layers. The feature set is transformed by the feature alignment module to obtain a transformed feature set, as shown in the following formula:

[0012]

[0013] Where V represents the feature set, W1 represents the weight of the first hidden layer in the feature alignment module, W2 represents the weight of the second hidden layer in the feature alignment module, norm(·) represents the batch normalization operation, σ(·) represents the relu activation function, represents the transformed feature set, v i represents the i-th feature in the converted feature set, i represents the feature number, when i = 0, v0 represents the converted global feature, when i = 1, 2, ..., N, v1, v2, ..., v N They represent the converted local features of the 1st, 2nd to Nth image blocks respectively.

[0014] Preferably, updating the last updated storage matrix according to the converted feature set of the current image to obtain the current updated storage matrix specifically includes:

[0015] The cosine similarity between each feature in the converted feature set of the current image and each cache entry of the last updated storage matrix is ​​calculated, and the probability is obtained through the softmax function, as shown in the following formula:

[0016]

[0017] Among them, D i，j represents the probability of assigning the i-th feature in the converted feature set of the current image to the j-th cache entry in the storage matrix A after the last update, exp(·) represents the exponential function, cos(·,·) represents the calculation of the cosine similarity between two vectors, and v i Represents the i-th feature in the converted feature set of the current image, A j represents the jth cache entry in the storage matrix A after the last update, j=1, 2, ..., M, M represents the total number of cache entries;

[0018] Each feature in the converted feature set of the current image is assigned to the cache entry with the highest probability in the last updated storage matrix A according to the probability, as shown in the following formula:

[0019] G j ={i|argmax k D i，k =j};

[0020] Among them, G j Represents the set of serial numbers of all features in the converted feature set of the current image that are assigned to the jth cache entry in the storage matrix A after the last update, argmax k D i，k Indicates taking D i，k k corresponding to the maximum value in;

[0021] Collect information from each feature assigned to the jth cache entry in the last updated storage matrix A and update the last updated storage matrix A, as shown in the following formula:

[0022]

[0023] Among them, v h represents one of the features assigned to the jth cache entry in the last updated storage matrix A, γ represents the momentum coefficient, A j represents the jth cache entry in the storage matrix A after the last update, A′ j Represents the jth cache entry in the storage matrix after the current update.

[0024] Preferably, the storage matrix and text embedding after the current update are input into the feature aggregation module to obtain the current aggregation features, which specifically include:

[0025] Match the text vector of the u-th category in the text embedding with each cache entry in the current updated storage matrix to obtain the matching weight, as shown in the following formula:

[0026]

[0027] Among them, E u Represents the text vector of the u-th category in the text embedding, W u，j Represents the matching weight between the text vector of the u-th category in the text embedding and the j-th cache entry in the storage matrix after the current update;

[0028] The detailed description embedding of each current category is calculated based on the matching weight and each cache entry in the storage matrix after the current update, as shown in the following formula:

[0029]

[0030] in, Represents the detailed descriptive embedding of the current u-th category;

[0031] The detailed description embedding of each current category is concatenated with the text vector of each category in the text embedding, and then aggregated using a linear layer and residual connection to obtain the aggregated features of each current category, as shown in the following formula:

[0032]

[0033] Among them, Linear(·) represents the function corresponding to the linear layer, α represents the hyperparameter, Represents the aggregated features of the current u-th category. The aggregated features of all current categories constitute the current aggregated features.

[0034] Preferably, the predicted label corresponding to the current image is calculated based on the current aggregated features and the global features of the current image, specifically including:

[0035] Calculate the similarity between the current aggregated features and the global features of the current image, and determine the predicted label based on the maximum likelihood, as shown in the following formula:

[0036]

[0037] Among them, y is the predicted label, represents the aggregated features of the current c-th category, C is the total number of categories, c = 1, 2, ..., C, τ represents the temperature coefficient.

[0038] In a second aspect, the present invention provides a few-sample fine-tuning device for a pre-trained visual language model, comprising:

[0039] A data acquisition module, configured to acquire a number of images, texts and true labels corresponding to each image and construct a fine-tuning dataset;

[0040] A model building module is configured to build a pre-trained visual language model, a feature alignment module, and a local cache module. The pre-trained visual language model includes an image encoder, a text encoder, and a feature aggregation module. The image encoder and the text encoder are respectively connected to the feature aggregation module.

[0041] The fine-tuning module is configured to input each image and text in the fine-tuning dataset into the pre-trained visual language model, divide each image into a number of non-overlapping image blocks through an image encoder, extract the global features of the image and the local features of each image block respectively and form a feature set of each image, input the feature set of each image into the feature alignment module, and obtain the converted feature set of each image; construct a storage matrix in the local cache module, randomly initialize the storage matrix to obtain an initial storage matrix, traverse the converted feature set of each image, and update the initial storage matrix according to the converted feature set of the first image. The storage matrix after the first update is obtained, and the storage matrix after the last update is updated according to the converted feature set of the current image to obtain the storage matrix after the current update; each text is input into the text encoder to obtain the text embedding, and the storage matrix after the current update and the text embedding are input into the feature aggregation module to obtain the current aggregated features, and the predicted label corresponding to the current image is calculated according to the current aggregated features and the global features of the current image, and the loss function is constructed using the predicted labels and the true labels. Based on the loss function, the pre-trained visual language model is fine-tuned to obtain the comparison between the pre-trained visual language model and the storage matrix after the last update.

[0042] In a third aspect, the present invention provides a method for image-text comparison prediction, using a comparison pre-trained visual language model obtained by the method described in any implementation of the first aspect and a storage matrix after the last update, comprising the following steps:

[0043] Get the image to be processed and the input text;

[0044] The image to be processed and the input text are compared with the pre-trained visual language model, and the image to be processed is divided into several non-overlapping image blocks through the image encoder, and the global features of the image to be processed are extracted respectively; the input text is input into the text encoder to obtain the text vector corresponding to the input text, and the storage matrix after the last update and the text vector corresponding to the input text are input into the feature aggregation module to obtain the aggregated features, and the predicted label corresponding to the image to be processed is calculated according to the aggregated features and the global features of the image to be processed.

[0045] In a fourth aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0046] In a fifth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0047] In a sixth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] (1) The few-shot fine-tuning method of the pre-trained visual language model proposed in the present invention introduces a local buffer module to encapsulate fine-grained visual concepts extracted from the global features and local features of the image branch. During the fine-tuning process of the pre-trained visual language model, the image branch will continuously write fine-grained semantic information into the local buffer module by category, and then the fine-grained semantic information will be spliced ​​with the corresponding text embedding, and then the global and local information will be fused through the feature aggregation module to enhance the representation ability of the text branch.

[0050] (2) The few-shot fine-tuning method for the pre-trained visual language model proposed in the present invention introduces a feature alignment module to convert global features and local features into the text embedding space, thereby alleviating the modality gap and solving the potential feature shift problem.

[0051] (3) The few-sample fine-tuning method of the pre-trained visual language model proposed in the present invention can achieve plug-and-play, refine the prompt text during tuning, and does not need to rely on the help of external experts. It can perform efficient transfer learning in low-data scenarios while maintaining the generalization ability on downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 A schematic diagram of a visual language model that implements local alignment with the help of an external expert knowledge base in the prior art;

[0054] Figure 2 A schematic diagram of a process flow of a few-sample fine-tuning method for a pre-trained visual language model according to an embodiment of the present application;

[0055] Figure 3 A schematic diagram of a framework of a few-sample fine-tuning method for a pre-trained visual language model according to an embodiment of the present application;

[0056] Figure 4 A schematic diagram of the reasoning process of a visual language model obtained by a few-sample fine-tuning method of a pre-trained visual language model according to an embodiment of the present application;

[0057] Figure 5 A diagram showing the comparison of the inference efficiency and performance of a visual language model obtained by a few-sample fine-tuning method of a pre-trained visual language model according to an embodiment of the present application with other methods;

[0058] Figure 6 A schematic diagram of a few-sample fine-tuning device for a pre-trained visual language model according to an embodiment of the present application;

[0059] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] Figure 2 A few-sample fine-tuning method for a pre-trained visual language model provided in an embodiment of the present application is shown, comprising the following steps:

[0062] S1, obtain several images, texts and the true labels corresponding to each image and build a fine-tuning dataset.

[0063] Specifically, the text input in the embodiment of the present application is a category prompt text in the format of "A photo of a[class]", which is used to prompt the category corresponding to the input image. The true label is the serial number of the true category corresponding to the image. A small amount of images, texts and true labels corresponding to the images are collected to construct a fine-tuning dataset.

[0064] S2, constructs a pre-trained visual language model, a feature alignment module and a local cache module. The pre-trained visual language model includes an image encoder, a text encoder and a feature aggregation module. The image encoder and the text encoder are connected to the feature aggregation module respectively.

[0065] Specifically, refer to Figure 3 , the fine-tuning process of the pre-trained visual language model is optimized by using a feature alignment module and a local cache module. The embodiment of the present application introduces a feature alignment module for aligning the global features and local features of the image, and further introduces a novel local cache module to encapsulate the fine-grained visual concepts extracted from the global features and local features of the image branch. During the migration process of the visual language model, the image branch will continuously write the fine-grained semantic information into the local cache module by category, and then these fine-grained semantic information will be spliced ​​with the corresponding text vector, and then the global information and local information will be fused through the feature aggregation module to enhance the representation ability of the text branch. In an embodiment of the present application, the image encoder adopts the ViT (Vision Transformer) model, which is an image classification network built based on the Transformer model, so the input image can be divided into non-overlapping image blocks, and all the image blocks are input into the stacked transformer blocks to obtain global features and a series of local features, and constitute the feature set V of the image.

[0066] S3, input each image and text in the fine-tuning dataset into the pre-trained visual language model, divide each image into several non-overlapping image blocks through the image encoder, extract the global features of the image and the local features of each image block respectively and form a feature set of each image, input the feature set of each image into the feature alignment module, and obtain the converted feature set of each image; construct a storage matrix in the local cache module, randomly initialize the storage matrix to obtain the initial storage matrix, traverse the converted feature set of each image, and update the initial storage matrix according to the converted feature set of the first image to obtain the first converted feature set. After an update, the storage matrix after the last update is updated according to the converted feature set of the current image to obtain the storage matrix after the current update; each text is input into the text encoder to obtain text embedding, and the storage matrix and text embedding after the current update are input into the feature aggregation module to obtain the current aggregated features, and the predicted label corresponding to the current image is calculated according to the current aggregated features and the global features of the current image, and the loss function is constructed using the predicted labels and the true labels. Based on the loss function, the pre-trained visual language model is fine-tuned to obtain a comparison between the pre-trained visual language model and the storage matrix after the last update.

[0067] In a specific embodiment, the feature alignment module is a multilayer perceptron with two hidden layers. The feature set is transformed by the feature alignment module to obtain a transformed feature set, as shown in the following formula:

[0068]

[0069] Where V represents the feature set, W1 represents the weight of the first hidden layer in the feature alignment module, W2 represents the weight of the second hidden layer in the feature alignment module, norm(·) represents the batch normalization operation, σ(·) represents the relu activation function, represents the transformed feature set, v i represents the i-th feature in the converted feature set, i represents the feature number, when i = v, v0 represents the converted global feature, when i = 1, 2, ..., N, v1, v2, ..., v N They represent the converted local features of the 1st, 2nd to Nth image blocks respectively.

[0070] Specifically, the pre-trained visual language model can align the global features of the image and text to learn the joint embedding space. However, the local features in the image lack clear alignment, resulting in a modal gap between the local features and the text embedding E. In order to solve the potential feature offset problem, an embodiment of the present application introduces a feature alignment module to convert the global features and local features of the image into the text embedding space, thereby alleviating the modal gap. The feature alignment module proposed in the embodiment of the present application consists of only a multi-layer perceptron with two hidden layers.

[0071] In a specific embodiment, updating the last updated storage matrix according to the converted feature set of the current image to obtain the current updated storage matrix specifically includes:

[0072] The cosine similarity between each feature in the converted feature set of the current image and each cache entry of the last updated storage matrix is ​​calculated, and the probability is obtained through the softmax function, as shown in the following formula:

[0073]

[0074] Among them, D i，j represents the probability of assigning the i-th feature in the converted feature set of the current image to the j-th cache entry in the storage matrix A after the last update, exp(·) represents the exponential function, cos(·,·) represents the calculation of the cosine similarity between two vectors, and v i Represents the i-th feature in the converted feature set of the current image, A j represents the jth cache entry in the storage matrix A after the last update, j=1, 2, ..., M, M represents the total number of cache entries;

[0075] Each feature in the converted feature set of the current image is assigned to the cache entry with the highest probability in the last updated storage matrix A according to the probability, as shown in the following formula:

[0076] G j ={i|argmax k D i，k =j};

[0077] Among them, G i Represents the set of serial numbers of all features in the converted feature set of the current image that are assigned to the jth cache entry in the storage matrix A after the last update, argmax k D i，k Indicates taking D i，k The k corresponding to the maximum value in is:

[0078] Collect information from each feature assigned to the jth cache entry in the last updated storage matrix A and update the last updated storage matrix A, as shown in the following formula:

[0079]

[0080] Among them, v n represents one of the features assigned to the jth cache entry in the last updated storage matrix A, γ represents the momentum coefficient, A j represents the jth cache entry in the storage matrix A after the last update, A′ j Represents the jth cache entry in the storage matrix after the current update.

[0081] Specifically, local features have been widely proven to be able to capture specialized visual features, such as edges, textures, etc., and represent recognizable conceptual categories. Therefore, the embodiments of the present application utilize these local features within the pre-trained visual language model as visual attribute descriptions of categories, abandoning the practice of using external experts in previous methods, and thus propose a local cache module for storing fine-grained visual concepts in global features and local features. The embodiments of the present application do not roughly store all global features and local features in batches, but instead establish a fixed number of cache entries and cluster similar global features and local features into one cache entry to selectively capture the corresponding visual attributes. For example, when processing zebra samples, the goal is to fuse its significant black and white stripe features into a cache entry representing texture. To this end, the embodiments of the present application construct a storage matrix A∈R M×d, where M represents the total number of cache entries, the M cache entries in the storage matrix are used to collect information from the global features and local features of all instances based on similarity, d represents the feature dimension, the row vector of the jth row in the storage matrix is ​​the jth cache entry in the storage matrix, and in the initial stage, the storage matrix is ​​randomly initialized to obtain the initial storage matrix. When the first image and text are input to the pre-trained visual language model, the first image passes through the image encoder and feature alignment module in sequence to obtain the converted feature set of the first image, and the initial storage matrix is ​​updated according to the converted feature set of the first image to obtain the first updated storage matrix. Specifically, it includes: calculating the cosine similarity between each cache entry in the initial storage matrix and each feature in the converted feature set of the first image, and then obtaining the probability through the softmax function. Then, each feature in the converted feature set of the first image is assigned to the cache entry with the highest probability in the initial storage matrix. Subsequently, each cache entry collects information from each feature in the converted feature set of the first image to update the initial storage matrix. After obtaining the first updated storage matrix, the previously updated storage matrix is ​​updated in the same manner through the converted feature set of the current image to obtain the current updated storage matrix. Through the above update, the embodiment of the present application realizes the continuous memory of the fine-grained information inherent in the global features and local features, which can then be used to enhance the descriptive ability of the text representation.

[0082] In a specific embodiment, the storage matrix and text embedding after the current update are input into the feature aggregation module to obtain the current aggregated features, which specifically include:

[0083] Match the text vector of the u-th category in the text embedding with each cache entry in the current updated storage matrix to obtain the matching weight, as shown in the following formula:

[0084]

[0085] Among them, E u Represents the text vector of the u-th category in the text embedding, W u，j Represents the matching weight between the text vector of the u-th category in the text embedding and the j-th cache entry in the storage matrix after the current update;

[0086] The detailed description embedding of each current category is calculated based on the matching weight and each cache entry in the storage matrix after the current update, as shown in the following formula:

[0087]

[0088] in, Represents the detailed descriptive embedding of the current u-th category;

[0089] The detailed description embedding of each current category is concatenated with the text vector of each category in the text embedding, and then aggregated using a linear layer and residual connection to obtain the aggregated features of each current category, as shown in the following formula:

[0090]

[0091] Among them, Linear(·) represents the function corresponding to the linear layer, α represents the hyperparameter, Represents the aggregated features of the current u-th category. The aggregated features of all current categories constitute the current aggregated features.

[0092] Specifically, in order to utilize the fine-grained information in global features and local features, an embodiment of the present application introduces a feature aggregation module, combines the context in local features with the global context in the text, and enhances the text embedding of categories through residual connections. Specifically, texts of different categories are input into the text encoder to obtain a matrix-type text embedding, and each row in the text embedding represents the text vector extracted by the text encoder from the text of the corresponding category. First, each text vector in the current text embedding from the text encoder is matched with each cache entry in the storage matrix after the current update, and the detailed description embedding related to each category is calculated, the detailed description embedding is concatenated with the original text embedding, and then a linear layer and a residual connection are used to aggregate these two features to obtain an aggregated feature.

[0093] In a specific embodiment, the prediction label corresponding to the current image is calculated based on the current aggregated features and the global features of the current image, specifically including:

[0094] Calculate the similarity between the current aggregated features and the global features of the current image, and determine the predicted label based on the maximum likelihood, as shown in the following formula:

[0095]

[0096] Among them, y is the predicted label, represents the aggregated features of the current c-th category, C is the total number of categories, c = 1, 2, ..., C, and τ represents the temperature coefficient.

[0097] Specifically, the cosine similarity between the global features of the current image and the aggregated features of all current categories is calculated, and the prediction probability is obtained through the softmax function, and the category number corresponding to the maximum prediction probability is selected as the prediction label.

[0098] The embodiment of the present application further proposes a method for image-text comparison prediction, characterized in that the comparison pre-trained visual language model and the storage matrix after the last update are obtained by using the few-sample fine-tuning method of the pre-trained visual language model of any one of claims 1 to 5, comprising the following steps:

[0099] Get the image to be processed and the input text;

[0100] The image to be processed and the input text are compared with the pre-trained visual language model, and the image to be processed is divided into several non-overlapping image blocks through the image encoder, and the global features of the image to be processed are extracted respectively; the input text is input into the text encoder to obtain the text vector corresponding to the input text, and the storage matrix after the last update and the text vector corresponding to the input text are input into the feature aggregation module to obtain the aggregated features, and the predicted label corresponding to the image to be processed is calculated according to the aggregated features and the global features of the image to be processed.

[0101] Specifically, refer to Figure 4 After the above-mentioned pre-trained visual language model is fine-tuned by a few samples, the pre-trained visual language model and the storage matrix after the last update are obtained, and then the inference stage can be entered. In the inference stage, the input image and text to be processed are input to the image encoder and the text encoder respectively to obtain the global features of the image to be processed and the text vector corresponding to the input text. The storage matrix after the last update and the text vector corresponding to the input text are feature aggregated by the feature aggregation module to obtain the aggregated features of all categories. The cosine similarity between the global features of the image to be processed and the aggregated features of all categories is calculated, and the prediction probability is obtained after the softmax function is passed. The category number corresponding to the maximum prediction probability is selected as the prediction label corresponding to the image to be processed.

[0102] Unlike other methods that insert hard-coded or carefully designed learnable hints (representing visual attributes) into the input embedding of the text branch, the comparative pre-trained visual language model (TextRefiner) proposed in the embodiment of the present application only needs to perform feature aggregation between the text vector and the local cache module, thereby providing efficiency advantages. Figure 5 As shown in the figure, FPS represents the number of images that can be processed per second. TextRefiner only slightly reduces FPS and still maintains high efficiency. In contrast, LLaMP, which relies on external experts, is significantly affected in inference speed.

[0103] Further references Figure 6 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a device for fine-tuning a few samples of a pre-trained visual language model. Figure 2Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0104] The embodiment of the present application provides a few-sample fine-tuning device for a pre-trained visual language model, including:

[0105] A data acquisition module 1 is configured to acquire a number of images, texts and true labels corresponding to each image and construct a fine-tuning dataset;

[0106] A model building module 2 is configured to build a pre-trained visual language model, wherein the pre-trained visual language model includes an image encoder, a text encoder, a feature alignment module, a feature aggregation module, and a local cache module;

[0107] The fine-tuning module 3 is configured to input each image and text in the fine-tuning data set into the pre-trained visual language model, divide each image into a number of non-overlapping image blocks through an image encoder, extract the global features of the image and the local features of each image block respectively and form a feature set of each image, input the feature set of each image into the feature alignment module, and obtain the converted feature set of each image; construct a storage matrix in the local cache module, randomly initialize the storage matrix to obtain an initial storage matrix, traverse the converted feature set of each image, and update the initial storage matrix according to the converted feature set of the first image. The storage matrix after the first update is obtained, and the storage matrix after the last update is updated according to the converted feature set of the current image to obtain the storage matrix after the current update; each text is input into the text encoder to obtain the text embedding, and the storage matrix after the current update and the text embedding are input into the feature aggregation module to obtain the current aggregated features, and the predicted label corresponding to the current image is calculated according to the current aggregated features and the global features of the current image, and the loss function is constructed using the predicted labels and the true labels. Based on the loss function, the pre-trained visual language model is fine-tuned to obtain the comparison between the pre-trained visual language model and the storage matrix after the last update.

[0108] Figure 7 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Figure 7 As shown, the electronic device of this embodiment includes: a processor 701 and a memory 702; wherein the memory 702 is used to store computer-executable instructions; the processor 701 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0109] Optionally, the memory 702 may be independent or integrated with the processor 701 .

[0110] When the memory 702 is independently provided, the electronic device further includes a bus 703 for connecting the memory 702 and the processor 701 .

[0111] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 701 executes the computer execution instructions, the above method is implemented.

[0112] The embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 701, the above method is implemented.

[0113] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0114] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0115] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The unit formed by the above modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0116] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 701 to perform some steps of the methods of various embodiments of the present application.

[0117] It should be understood that the processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 701 may be any conventional processor 701, etc. The steps of the method disclosed in the invention may be directly embodied in the execution of the hardware processor 701, or may be executed by a combination of hardware and software modules in the processor 701.

[0118] The memory 702 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0119] The bus 703 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 703 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 703 in the drawings of the present application is not limited to only one bus 703 or one type of bus 703.

[0120] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0121] An exemplary storage medium is coupled to the processor 701, so that the processor 701 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 701. The processor 701 and the storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as ASIC). Of course, the processor 701 and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0122] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A few-shot fine-tuning method for a pre-trained visual language model, characterized in that: The following steps are involved: Get several images, texts and the true labels corresponding to each image and build a fine-tuning dataset; Constructing a pre-trained visual language model, a feature alignment module and a local cache module, wherein the pre-trained visual language model includes an image encoder, a text encoder and a feature aggregation module, wherein the image encoder and the text encoder are respectively connected to the feature aggregation module; Input each image and text in the fine-tuning dataset into the pre-trained visual language model, divide each image into a number of non-overlapping image blocks through the image encoder, extract the global features of the image and the local features of each image block respectively and form a feature set of each image, input the feature set of each image into the feature alignment module, and obtain the converted feature set of each image; construct a storage matrix in the local cache module, randomly initialize the storage matrix to obtain an initial storage matrix, traverse the converted feature set of each image, update the initial storage matrix according to the converted feature set of the first image, and obtain the first updated storage matrix, and update the last updated storage matrix according to the converted feature set of the current image to obtain the current updated storage matrix; Each text is input into the text encoder to obtain text embedding, the currently updated storage matrix and the text embedding are input into the feature aggregation module to obtain the current aggregated features, the predicted label corresponding to the current image is calculated according to the current aggregated features and the global features of the current image, a loss function is constructed using the predicted label and the true label, and the pre-trained visual language model is fine-tuned based on the loss function to obtain a comparison of the pre-trained visual language model and the storage matrix after the last update.

2. The method for fine-tuning a pre-trained visual language model with a few samples according to claim 1, characterized in that: The feature alignment module is a multi-layer perceptron with two hidden layers. The feature set is transformed by the feature alignment module to obtain a transformed feature set, as shown in the following formula: Where V represents the feature set, W1 represents the weight of the first hidden layer in the feature alignment module, W2 represents the weight of the second hidden layer in the feature alignment module, norm(·) represents the batch normalization operation, σ(·) represents the relu activation function, represents the transformed feature set, v i represents the i-th feature in the converted feature set, i represents the feature number, when i=0, v0 represents the converted global feature, when i=1,2,…,N, v1,v2,…,v N They represent the converted local features of the 1st, 2nd to Nth image blocks respectively.

3. The method for fine-tuning a pre-trained visual language model with a few samples according to claim 1, characterized in that: The storage matrix after the last update is updated according to the converted feature set of the current image to obtain the storage matrix after the current update, which specifically includes: The cosine similarity between each feature in the converted feature set of the current image and each cache entry of the last updated storage matrix is ​​calculated, and the probability is obtained through the softmax function, as shown in the following formula: Among them, D i,j represents the probability of assigning the i-th feature in the converted feature set of the current image to the j-th cache entry in the storage matrix A after the last update, exp(·) represents the exponential function, cos(·,·) represents the calculation of the cosine similarity between two vectors, and v i Represents the i-th feature in the converted feature set of the current image, A j represents the jth cache entry in the storage matrix A after the last update, j = 1, 2, ..., M, M represents the total number of cache entries; Each feature in the converted feature set of the current image is assigned to the cache entry with the highest probability in the last updated storage matrix A according to the probability, as shown in the following formula: G j ={i|argmax k D i,k =j}; Among them, G j Represents the set of serial numbers of all features in the converted feature set of the current image that are assigned to the jth cache entry in the storage matrix A after the last update, argmax k D i,k Indicates taking D i,k k corresponding to the maximum value in; Collect information from each feature assigned to the jth cache entry in the last updated storage matrix A and update the last updated storage matrix A, as shown in the following formula: Among them, v h represents one of the features assigned to the jth cache entry in the last updated storage matrix A, γ represents the momentum coefficient, A j represents the jth cache entry in the storage matrix A after the last update, A′ j Represents the jth cache entry in the storage matrix after the current update.

4. The method for fine-tuning a pre-trained visual language model with a few samples according to claim 3, characterized in that: Inputting the currently updated storage matrix and the text embedding into the feature aggregation module to obtain the current aggregated features specifically includes: The text vector of the u-th category in the text embedding is matched with each cache entry in the storage matrix after the current update to obtain a matching weight, as shown in the following formula: Among them, E u Represents the text vector of the u-th category in the text embedding, W u,j Represents the matching weight between the text vector of the u-th category in the text embedding and the j-th cache entry in the storage matrix after the current update; The detailed description embedding of each current category is calculated based on the matching weight and each cache entry in the storage matrix after the current update, as shown in the following formula: in, Represents the detailed descriptive embedding of the current u-th category; The detailed description embedding of each current category is concatenated with the text vector of each category in the text embedding, and then aggregated using a linear layer and residual connection to obtain the aggregated features of each current category, as shown in the following formula: Among them, Linear(·) represents the function corresponding to the linear layer, α represents the hyperparameter, Represents the aggregated features of the current u-th category. The aggregated features of all current categories constitute the current aggregated features.

5. The method for fine-tuning a pre-trained visual language model with a few samples according to claim 4, characterized in that: The predicted label corresponding to the current image is calculated based on the current aggregated features and the global features of the current image, specifically including: Calculate the similarity between the current aggregated features and the global features of the current image, and determine the predicted label based on the maximum likelihood, as shown in the following formula: Among them, y is the predicted label, represents the aggregated features of the current c-th category, C is the total number of categories, c = 1, 2, ..., C, and τ represents the temperature coefficient.

6. A few-sample fine-tuning device for a pre-trained visual language model, characterized in that: include: A data acquisition module, configured to acquire a number of images, texts and true labels corresponding to each image and construct a fine-tuning dataset; A model building module is configured to build a pre-trained visual language model, a feature alignment module, and a local cache module, wherein the pre-trained visual language model includes an image encoder, a text encoder, and a feature aggregation module, wherein the image encoder and the text encoder are respectively connected to the feature aggregation module; A fine-tuning module is configured to input each image and text in the fine-tuning data set into the pre-trained visual language model, divide each image into a number of non-overlapping image blocks through the image encoder, extract the global features of the image and the local features of each image block respectively and form a feature set of each image, input the feature set of each image into the feature alignment module, and obtain the converted feature set of each image; construct a storage matrix in the local cache module, randomly initialize the storage matrix to obtain an initial storage matrix, traverse the converted feature set of each image, update the initial storage matrix according to the converted feature set of the first image, and obtain the first updated storage matrix, and update the last updated storage matrix according to the converted feature set of the current image to obtain the current updated storage matrix; Each text is input into the text encoder to obtain text embedding, the currently updated storage matrix and the text embedding are input into the feature aggregation module to obtain the current aggregated features, the predicted label corresponding to the current image is calculated according to the current aggregated features and the global features of the current image, a loss function is constructed using the predicted label and the true label, and the pre-trained visual language model is fine-tuned based on the loss function to obtain a comparison of the pre-trained visual language model and the storage matrix after the last update.

7. A method for predicting image-text comparison, characterized in that: The method for fine-tuning the pre-trained visual language model according to any one of claims 1 to 5 is used to obtain a comparison between the pre-trained visual language model and the storage matrix after the last update, comprising the following steps: Get the image to be processed and the input text; The image to be processed and the input text are input into the comparative pre-trained visual language model, the image to be processed is divided into a plurality of non-overlapping image blocks by the image encoder, and the global features of the image to be processed are extracted respectively; the input text is input into the text encoder to obtain a text vector corresponding to the input text, the storage matrix after the last update and the text vector corresponding to the input text are input into the feature aggregation module to obtain aggregated features, and the predicted label corresponding to the image to be processed is calculated according to the aggregated features and the global features of the image to be processed.

8. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images