A few-shot fine-tuning method and device of a pre-trained visual language model and readable medium

Abstract

The application discloses a few-shot fine-tuning method and device of a pre-trained visual language model and readable medium, comprising: constructing a fine-tuning data set, a pre-trained visual language model and a feature alignment module and a local cache module, the pre-trained visual language model comprising an image encoder, a text encoder and a feature aggregation module; fine-tuning the pre-trained visual language model based on the fine-tuning data set and through the feature alignment module and the local cache module, obtaining a contrast pre-trained visual language model and a last updated storage matrix. First, the global feature and the local feature are converted to a text embedding space through the feature comparison module, and the fine-grained semantic information extracted from the local feature of the image branch is encapsulated through the local cache module, then the fine-grained semantic information is spliced with the corresponding text embedding, and then the global and local information is fused through the feature aggregation module to enhance the representation ability of the text branch.

Description

Technical Field

[0001] This invention relates to the field of image and text processing, and specifically to a method, apparatus, and readable medium for fine-tuning a pre-trained visual language model using few samples. Background Technology

[0002] Traditional computer vision systems rely entirely on predefined label sets for visual representation learning, limiting their ability to scale to unseen categories. Visual language models overcome this limitation by introducing natural language supervision, enabling vision systems to grasp more visual concepts. These models are pre-trained in a self-supervised manner on large amounts of raw text and image pairing data obtained from the internet, optimizing the image and text encoders using contrastive loss to align text and image features into a unified embedding space. Thanks to large-scale datasets and contrastive training objectives, the visual representations learned by visual language models are connected to language and possess high generalization ability, making them widely applicable to various downstream vision tasks such as object detection, semantic segmentation, and depth estimation. Despite these powerful capabilities, transferring these visual language models to downstream tasks in low-sample scenarios still faces significant challenges. The substantial reduction in data volume compared to the pre-training stage typically leads to a significant decrease in the generalization ability of visual language models on transfer tasks.

[0003] To date, much research has focused on transferring visual language models with limited sample sizes. The main focus has been on using cue learning, a well-established strategy for adapting large pre-trained models because it avoids completely retraining the original model. Instead, it achieves downstream transfer by introducing learnable textual or visual cues that go beyond manually defined cues in the pre-trained model, such as "a photo of a {class}".

[0004] As an initial exploration, CoOp incorporates learnable labels into predefined text prompts in the text branch. These labels are shared across all categories, acquiring task-related knowledge from downstream datasets to align with image features. CoCoOp constrains image features in an instance-specific manner to reduce overfitting in few-shot scenarios and enhance generalization to unseen categories. PromptSRC further leverages pre-trained features to tune the learned prompts for enhanced generalization. Subsequent research efforts have attempted to extend prompt learning to multimodal branches; for example, MaPLe extends prompt learning to both image and text branches, enabling prompts to capture features not only in the text branch but also enhance intermodal alignment.

[0005] Despite ongoing progress, these works share a common limitation: they tend to enhance high-level semantics in a coarse-grained, global manner, thus failing to achieve overall alignment across modalities. Consequently, customized cues may not guide models to recognize diverse visual concepts from local regions, limiting their generalization ability across categories with similar visual attributes. To overcome this limitation, recent research, such as ArGue and LLaMP, has enriched text cues with large language models (LLMs) by incorporating fine-grained category descriptions to improve transfer performance, such as... Figure 1 As shown. Unfortunately, these methods require additional steps to filter the outputs of LLMs to ensure their relevance to downstream datasets, thus introducing significant inference overhead. Summary of the Invention

[0006] The purpose of this application is to provide a few-sample fine-tuning method, apparatus, and readable medium for a pre-trained visual language model to address the aforementioned technical problems.

[0007] In a first aspect, the present invention provides a few-shot fine-tuning method for a pre-trained visual language model, comprising the following steps:

[0008] Obtain several images, text, and the corresponding ground truth labels for each image, and construct a fine-tuning dataset;

[0009] A pre-trained visual language model, a feature alignment module, and a local caching module are constructed. The pre-trained visual language model includes an image encoder, a text encoder, and a feature aggregation module. The image encoder and the text encoder are connected to the feature aggregation module, respectively.

[0010] Each image and text in the fine-tuning dataset is input into a pre-trained visual language model. An image encoder divides each image into several non-overlapping image patches, extracting global features and local features from each patch to form a feature set for each image. This feature set is then input into a feature alignment module to obtain a transformed feature set for each image. A storage matrix is ​​constructed in a local caching module and randomly initialized to obtain an initial storage matrix. The transformed feature sets of each image are iterated through, and the initial storage matrix is ​​updated based on the transformed feature set of the first image to obtain the first updated storage matrix. The previously updated storage matrix is ​​then updated based on the transformed feature set of the current image to obtain the current updated storage matrix. Each text is input into a text encoder to obtain a text embedding. The current updated storage matrix and the text embedding are input into a feature aggregation module to obtain the current aggregated features. The predicted label for the current image is calculated based on the current aggregated features and the global features of the current image. A loss function is constructed using the predicted label and the ground truth label. The pre-trained visual language model is then fine-tuned based on the loss function, resulting in a comparison between the pre-trained visual language model and the last updated storage matrix.

[0011] Preferably, the feature alignment module is a multilayer perceptron with two hidden layers. The feature set is transformed by the feature alignment module to obtain the transformed feature set, as shown in the following formula:

[0012]

[0013] Where V represents the feature set, W1 represents the weights of the first hidden layer in the feature alignment module, W2 represents the weights of the second hidden layer in the feature alignment module, norm(·) represents the batch normalization operation, and σ(·) represents the ReLU activation function. Represents the transformed feature set. v i Let vi represent the i-th feature in the transformed feature set, where i represents the feature index. When i = 0, v0 represents the transformed global feature; when i = 1, 2, ..., N, vi ... N These represent the transformed local features of the 1st, 2nd to Nth image patches, respectively.

[0014] Preferably, the storage matrix after the previous update is updated based on the transformed feature set of the current image to obtain the storage matrix after the current update, specifically including:

[0015] Calculate the cosine similarity between each feature in the transformed feature set of the current image and each cached entry in the previously updated storage matrix, and obtain the probability using the softmax function, as shown in the following formula:

[0016]

[0017] Among them, D i,j This represents the probability of assigning the i-th feature from the transformed feature set of the current image to the j-th cache entry in the previously updated storage matrix A. exp(·) represents the exponential function, and cos(·,·) represents calculating the cosine similarity between two vectors. i Let A represent the i-th feature in the transformed feature set of the current image. j This represents the j-th cache entry in the storage matrix A after the last update, where j = 1, 2, ..., M, and M represents the total number of cache entries;

[0018] Each feature in the transformed feature set of the current image is assigned to the cache entry with the highest probability in the previously updated storage matrix A, as shown in the following formula:

[0019] G j ={i|argmax k D i,k =j};

[0020] Among them, G j argmax represents the set of indices of all features in the transformed feature set of the current image that are assigned to the position of the j-th cache entry in the previously updated storage matrix A. k D i,k Indicates taking D i,k The maximum value corresponding to k in the range;

[0021] Information is collected from each feature of the j-th cache entry assigned to the last updated storage matrix A, and the last updated storage matrix A is updated as follows:

[0022]

[0023] Among them, v h This represents one of the features assigned to the position of the j-th cache entry in the last updated storage matrix A, where γ represents the momentum coefficient, and A j Let A' represent the j-th cache entry in the previously updated storage matrix A. j This represents the j-th cache entry in the storage matrix after the current update.

[0024] Preferably, the updated storage matrix and text embedding are input into the feature aggregation module to obtain the current aggregated features, which specifically include:

[0025] The matching weights are obtained by matching the text vector of the u-th category in the text embedding with each cached entry in the updated storage matrix, as shown in the following formula:

[0026]

[0027] Among them, E u W represents the text vector of the u-th category in the text embedding. u,j This represents the matching weight between the text vector of the u-th category in the text embedding and the j-th cache entry in the currently updated storage matrix;

[0028] The detailed embedding of each category is calculated based on the matching weight and each cached entry in the current updated storage matrix, as shown in the following formula:

[0029]

[0030] in, This represents the detailed embedding of the current u-th category;

[0031] The detailed description embedding of each current category is concatenated with the text vector of each category in the text embedding, and then aggregated using a linear layer and residual connections to obtain the aggregated features of each current category, as shown in the following formula:

[0032]

[0033] Where Linear(·) represents the function corresponding to the linear layer, and α represents the hyperparameter. This represents the aggregated feature of the current u-th category. The aggregated features of all categories constitute the current aggregated feature.

[0034] Preferably, the predicted label for the current image is calculated based on the current aggregated features and the global features of the current image, specifically including:

[0035] Calculate the similarity between the current aggregated features and the global features of the current image, and determine the predicted label based on the maximum likelihood, as shown in the following formula:

[0036]

[0037] Where y is the predicted label. This represents the aggregation feature of the current c-th category, where C is the total number of categories, c = 1, 2, ..., C, and τ represents the temperature coefficient.

[0038] Secondly, the present invention provides a few-shot fine-tuning device for a pre-trained visual language model, comprising:

[0039] The data acquisition module is configured to acquire several images, text, and the corresponding real labels for each image, and to construct a fine-tuning dataset;

[0040] The model building module is configured to build a pre-trained visual language model, as well as a feature alignment module and a local caching module. The pre-trained visual language model includes an image encoder, a text encoder, and a feature aggregation module. The image encoder and the text encoder are connected to the feature aggregation module, respectively.

[0041] The fine-tuning module is configured to input each image and text from the fine-tuning dataset into a pre-trained visual language model. It divides each image into several non-overlapping image patches using an image encoder, extracts global features from each image and local features from each patch, and constructs a feature set for each image. This feature set is then input into a feature alignment module to obtain the transformed feature set for each image. A storage matrix is ​​constructed in the local caching module, randomly initialized, and iterates through the transformed feature sets of each image, updating the initial storage matrix based on the transformed feature set of the first image. The storage matrix after the first update is obtained. The storage matrix after the previous update is updated based on the transformed feature set of the current image to obtain the storage matrix after the current update. Each text is input into the text encoder to obtain the text embedding. The storage matrix after the current update and the text embedding are input into the feature aggregation module to obtain the current aggregated features. The predicted label corresponding to the current image is calculated based on the current aggregated features and the global features of the current image. The loss function is constructed using the predicted label and the true label. The pre-trained visual language model is fine-tuned based on the loss function to obtain a comparison between the pre-trained visual language model and the storage matrix after the last update.

[0042] Thirdly, the present invention provides a method for image-text comparison prediction, which uses a comparison pre-trained visual language model and a last updated storage matrix obtained by the method described in any implementation of the first aspect, and includes the following steps:

[0043] Obtain the image to be processed and the input text;

[0044] The image to be processed and the input text are compared with a pre-trained visual language model. The image to be processed is divided into several non-overlapping image blocks by an image encoder, and the global features of the image to be processed are extracted from each block. The input text is input into a text encoder to obtain the text vector corresponding to the input text. The last updated storage matrix and the text vector corresponding to the input text are input into a feature aggregation module to obtain aggregated features. The predicted label corresponding to the image to be processed is calculated based on the aggregated features and the global features of the image to be processed.

[0045] Fourthly, the present invention provides an electronic device including one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0046] Fifthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0047] In a sixth aspect, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in any of the implementations in the first aspect.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] (1) The few-shot fine-tuning method for the pre-trained visual language model proposed in this invention introduces a local buffer module to encapsulate fine-grained visual concepts extracted from the global and local features of the image branch. During the fine-tuning process of the pre-trained visual language model, the image branch continuously writes fine-grained semantic information into the local buffer module according to the category. Then, this fine-grained semantic information is concatenated with the corresponding text embedding. Subsequently, the global and local information are fused through the feature aggregation module to enhance the representation ability of the text branch.

[0050] (2) The few-sample fine-tuning method for pre-trained visual language models proposed in this invention introduces a feature alignment module, which transforms global and local features into the text embedding space, thereby reducing modality gap and solving potential feature offset problems.

[0051] (3) The few-sample fine-tuning method for pre-trained visual language models proposed in this invention can achieve plug-and-play functionality, refine prompt text during optimization, and does not require the help of external experts. It can perform efficient transfer learning in low-data scenarios while maintaining generalization ability on downstream tasks. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a schematic diagram of a visual language model that uses an external expert knowledge base to achieve local alignment in existing technologies.

[0054] Figure 2 This is a flowchart illustrating a few-shot fine-tuning method for a pre-trained visual language model, as described in an embodiment of this application.

[0055] Figure 3 This is a schematic diagram illustrating the framework of a few-shot fine-tuning method for a pre-trained visual language model, as described in an embodiment of this application.

[0056] Figure 4 This is a schematic diagram illustrating the reasoning process of a visual language model obtained by a few-shot fine-tuning method for a pre-trained visual language model, as described in an embodiment of this application.

[0057] Figure 5 The figure shows a comparison of the inference efficiency and performance of the visual language model obtained by the few-shot fine-tuning method of the pre-trained visual language model in the embodiments of this application with other methods.

[0058] Figure 6 A schematic diagram of a few-shot fine-tuning device for a pre-trained visual language model, as described in an embodiment of this application;

[0059] Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0061] Figure 2 This application illustrates a few-shot fine-tuning method for a pre-trained visual language model, comprising the following steps:

[0062] S1: Obtain several images, texts, and the corresponding real labels for each image, and construct a fine-tuning dataset.

[0063] Specifically, in the embodiments of this application, the input text is a category hint text, formatted as "Aphoto of a[class]", which indicates the category corresponding to the input image. The ground truth label is the ordinal number of the ground truth category corresponding to the image. A fine-tuning dataset is constructed by collecting a small number of images, text, and the ground truth labels corresponding to the images.

[0064] S2, construct a pre-trained visual language model, as well as a feature alignment module and a local caching module. The pre-trained visual language model includes an image encoder, a text encoder, and a feature aggregation module. The image encoder and the text encoder are connected to the feature aggregation module, respectively.

[0065] For details, please refer to Figure 3 This application optimizes the fine-tuning process of a pre-trained visual language model using a feature alignment module and a local caching module. An embodiment of this application introduces a feature alignment module to align global and local features of an image, and further introduces a novel local caching module to encapsulate fine-grained visual concepts extracted from the global and local features of image branches. During the transfer process of the visual language model, the image branch continuously writes fine-grained semantic information into the local caching module according to categories. This fine-grained semantic information is then concatenated with the corresponding text vectors, and subsequently, a feature aggregation module fuses the global and local information to enhance the representational power of the text branch. In the embodiments of this application, the image encoder uses the ViT (Vision Transformer) model, which is based on...

[0066] The Transformer model is used to build an image classification network. Therefore, the input image can be divided into non-overlapping image blocks, and all image blocks are input into stacked Transformer blocks to obtain global features and a series of local features, which constitute the feature set V of the image.

[0067] S3: Input each image and text from the fine-tuning dataset into the pre-trained visual language model. The image encoder divides each image into several non-overlapping image patches. Global features of the image and local features of each patch are extracted to form the feature set for each image. This feature set is then input into the feature alignment module to obtain the transformed feature set for each image. A storage matrix is ​​constructed in the local caching module and randomly initialized to obtain the initial storage matrix. The transformed feature sets of each image are iterated through, and the initial storage matrix is ​​updated based on the transformed feature set of the first image to obtain the final feature set. The updated storage matrix is ​​updated based on the transformed feature set of the current image to obtain the current updated storage matrix. Each text is input into the text encoder to obtain the text embedding. The current updated storage matrix and the text embedding are input into the feature aggregation module to obtain the current aggregated features. The predicted label corresponding to the current image is calculated based on the current aggregated features and the global features of the current image. A loss function is constructed using the predicted label and the true label. The pre-trained visual language model is fine-tuned based on the loss function to obtain a comparison between the pre-trained visual language model and the last updated storage matrix.

[0068] In a specific embodiment, the feature alignment module is a multilayer perceptron with two hidden layers. The feature set is transformed by the feature alignment module to obtain the transformed feature set, as shown in the following formula:

[0069]

[0070] Where V represents the feature set, W1 represents the weights of the first hidden layer in the feature alignment module, W2 represents the weights of the second hidden layer in the feature alignment module, norm(·) represents the batch normalization operation, and σ(·) represents the ReLU activation function. Represents the transformed feature set. v i Let vi represent the i-th feature in the transformed feature set, where i represents the feature index. When i = 0, v0 represents the transformed global feature; when i = 1, 2, ..., N, vi ... N These represent the transformed local features of the 1st, 2nd to Nth image patches, respectively.

[0071] Specifically, pre-trained visual language models can align global features of images and text to learn a joint embedding space. However, local features in images lack explicit alignment, leading to a modal gap between local features and the text embedding E. To address the potential feature shift problem, embodiments of this application introduce a feature alignment module that transforms global and local features of the image into the text embedding space, thereby mitigating the modal gap. The feature alignment module proposed in embodiments of this application consists of only a multilayer perceptron with two hidden layers.

[0072] In a specific embodiment, the storage matrix updated in the previous step is updated based on the transformed feature set of the current image to obtain the storage matrix updated in the current step, specifically including:

[0073] Calculate the cosine similarity between each feature in the transformed feature set of the current image and each cached entry in the previously updated storage matrix, and obtain the probability using the softmax function, as shown in the following formula:

[0074]

[0075] Among them, D i,j This represents the probability of assigning the i-th feature from the transformed feature set of the current image to the j-th cache entry in the previously updated storage matrix A. exp(·) represents the exponential function, and cos(·,·) represents calculating the cosine similarity between two vectors. i Let A represent the i-th feature in the transformed feature set of the current image. j This represents the j-th cache entry in the storage matrix A after the last update, where j = 1, 2, ..., M, and M represents the total number of cache entries;

[0076] Each feature in the transformed feature set of the current image is assigned to the cache entry with the highest probability in the previously updated storage matrix A, as shown in the following formula:

[0077] G j ={i|argmax k D i,k =j};

[0078] Among them, G j argmax represents the set of indices of all features in the transformed feature set of the current image that are assigned to the position of the j-th cache entry in the previously updated storage matrix A. k D i,k Indicates taking D i,k The maximum value corresponding to k in the range;

[0079] Information is collected from each feature of the j-th cache entry assigned to the last updated storage matrix A, and the last updated storage matrix A is updated as follows:

[0080]

[0081] Among them, v h This represents one of the features assigned to the position of the j-th cache entry in the last updated storage matrix A, where γ represents the momentum coefficient, and A j Let A' represent the j-th cache entry in the previously updated storage matrix A. j This represents the j-th cache entry in the storage matrix after the current update.

[0082] Specifically, local features have been widely proven to capture specialized visual features, such as edges and textures, and represent identifiable concept categories. Therefore, embodiments of this application utilize these local features within a pre-trained visual language model as descriptions of category visual attributes, abandoning the previous practice of using external experts. This leads to the proposal of a local caching module for storing fine-grained visual concepts from global and local features. Instead of coarsely batch-storing all global and local features, embodiments of this application establish a fixed number of cache entries and cluster similar global and local features into a single cache entry to selectively capture corresponding visual attributes. For example, when processing zebra samples, the goal is to fuse their prominent black and white stripe features into a single cache entry representing texture. To this end, embodiments of this application construct a storage matrix A∈R M×dIn this context, M represents the total number of cache entries. The M cache entries in the storage matrix are used to collect information from the global and local features of all instances based on similarity. d represents the feature dimension, and the row vector of the j-th row in the storage matrix is ​​the j-th cache entry. In the initial stage, the storage matrix is ​​randomly initialized to obtain an initial storage matrix. When the first image and text are input into the pre-trained visual language model, the first image passes through the image encoder and feature alignment module sequentially to obtain the transformed feature set of the first image. The initial storage matrix is ​​updated based on the transformed feature set of the first image to obtain the first updated storage matrix. Specifically, this includes calculating the cosine similarity between each cache entry in the initial storage matrix and each feature in the transformed feature set of the first image, and then obtaining the probability through the softmax function. Then, each feature in the transformed feature set of the first image is assigned to the cache entry with the highest probability in the initial storage matrix. Subsequently, each cached entry collects information from each feature in the transformed feature set of the first image to update the initial storage matrix. After obtaining the first updated storage matrix, the previously updated storage matrix is ​​updated in the same way using the transformed feature set of the current image to obtain the current updated storage matrix. Through the above updates, the embodiments of this application achieve continuous memorization of fine-grained information inherent in global and local features, which can then be used to enhance the descriptive capabilities of text representations.

[0083] In a specific embodiment, the updated storage matrix and text embedding are input into the feature aggregation module to obtain the current aggregated features, specifically including:

[0084] The matching weights are obtained by matching the text vector of the u-th category in the text embedding with each cached entry in the updated storage matrix, as shown in the following formula:

[0085]

[0086] Among them, E u W represents the text vector of the u-th category in the text embedding. u,j This represents the matching weight between the text vector of the u-th category in the text embedding and the j-th cache entry in the currently updated storage matrix;

[0087] The detailed embedding of each category is calculated based on the matching weight and each cached entry in the current updated storage matrix, as shown in the following formula:

[0088]

[0089] in, This represents the detailed embedding of the current u-th category;

[0090] The detailed description embedding of each current category is concatenated with the text vector of each category in the text embedding, and then aggregated using a linear layer and residual connections to obtain the aggregated features of each current category, as shown in the following formula:

[0091]

[0092] Where Linear(·) represents the function corresponding to the linear layer, and α represents the hyperparameter. This represents the aggregated feature of the current u-th category. The aggregated features of all categories constitute the current aggregated feature.

[0093] Specifically, to utilize fine-grained information from both global and local features, embodiments of this application introduce a feature aggregation module. This module combines the context from local features with the global context from the text and enhances the text embeddings of categories through residual connections. Specifically, text from different categories is input into a text encoder to obtain matrix-type text embeddings. Each row in the text embedding represents a text vector extracted by the text encoder from the text of the corresponding category. First, each text vector in the current text embedding from the text encoder is matched with each cached entry in the updated storage matrix to calculate a detailed description embedding associated with each category. This detailed description embedding is then concatenated with the original text embedding. Finally, linear layers and residual connections are used to aggregate these two types of features, resulting in aggregated features.

[0094] In a specific embodiment, the predicted label corresponding to the current image is calculated based on the current aggregated features and the global features of the current image, specifically including:

[0095] Calculate the similarity between the current aggregated features and the global features of the current image, and determine the predicted label based on the maximum likelihood, as shown in the following formula:

[0096]

[0097] Where y is the predicted label. This represents the aggregation feature of the current c-th category, where C is the total number of categories, c = 1, 2, ..., C, and τ represents the temperature coefficient.

[0098] Specifically, the cosine similarity between the global features of the current image and the aggregated features of all current categories is calculated, and then passed through the softmax function to obtain the prediction probability. The category number corresponding to the highest prediction probability is selected as the prediction label.

[0099] The embodiments of this application also propose a text-image comparison prediction method, which uses the aforementioned few-shot fine-tuning method of the pre-trained visual language model to compare the pre-trained visual language model with the last updated storage matrix, including the following steps:

[0100] Obtain the image to be processed and the input text;

[0101] The image to be processed and the input text are compared with a pre-trained visual language model. The image to be processed is divided into several non-overlapping image blocks by an image encoder, and the global features of the image to be processed are extracted from each block. The input text is input into a text encoder to obtain the text vector corresponding to the input text. The last updated storage matrix and the text vector corresponding to the input text are input into a feature aggregation module to obtain aggregated features. The predicted label corresponding to the image to be processed is calculated based on the aggregated features and the global features of the image to be processed.

[0102] For details, please refer to Figure 4 After the few-shot fine-tuning method of the pre-trained visual language model described above is used to obtain a comparison between the pre-trained visual language model and the last updated storage matrix, the inference stage can begin. In the inference stage, the input image and text are fed into the image encoder and text encoder, respectively, to obtain the global features of the image and the corresponding text vector of the input text. The feature aggregation module aggregates the features of the last updated storage matrix and the corresponding text vector of the input text to obtain aggregated features for all categories. The cosine similarity between the global features of the image and the aggregated features for all categories is calculated and passed through a softmax function to obtain the prediction probability. The category number corresponding to the highest prediction probability is selected as the predicted label for the image.

[0103] Unlike other methods that insert hard-coded or carefully crafted learnable cues (representing visual attributes) into the input embedding of text branches, the contrast-pre-trained visual language model (TextRefiner) proposed in this application only requires feature aggregation between text vectors and local cache modules, thus providing an efficiency advantage. Figure 5 As shown, FPS represents the number of images that can be processed per second. TextRefiner only slightly reduces the FPS, still maintaining high efficiency. In contrast, LLaMP, which relies on external experts, is significantly affected in inference speed.

[0104] Further reference Figure 6 As an implementation of the methods shown in the above figures, this application provides an embodiment of a few-shot fine-tuning device for a pre-trained visual language model, which is similar to... Figure 2Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0105] This application provides a few-shot fine-tuning device for a pre-trained visual language model, comprising:

[0106] Data acquisition module 1 is configured to acquire several images, text, and the real labels corresponding to each image, and construct a fine-tuning dataset;

[0107] Model building module 2 is configured to build a pre-trained visual language model, which includes an image encoder, a text encoder, a feature alignment module, a feature aggregation module, and a local caching module.

[0108] Fine-tuning module 3 is configured to input each image and text from the fine-tuning dataset into a pre-trained visual language model. It divides each image into several non-overlapping image patches using an image encoder, extracts global features from each image and local features from each patch, and constructs a feature set for each image. This feature set is then input into a feature alignment module to obtain the transformed feature set for each image. A storage matrix is ​​constructed in the local caching module, randomly initialized, and iterates through the transformed feature sets of each image, updating the initial storage matrix based on the transformed feature set of the first image. The storage matrix after the first update is obtained. The storage matrix after the previous update is updated based on the transformed feature set of the current image to obtain the storage matrix after the current update. Each text is input into the text encoder to obtain the text embedding. The storage matrix after the current update and the text embedding are input into the feature aggregation module to obtain the current aggregated features. The predicted label corresponding to the current image is calculated based on the current aggregated features and the global features of the current image. The loss function is constructed using the predicted label and the true label. The pre-trained visual language model is fine-tuned based on the loss function to obtain a comparison between the pre-trained visual language model and the storage matrix after the last update.

[0109] Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. For example... Figure 7 As shown, the electronic device of this embodiment includes a processor 701 and a memory 702; wherein the memory 702 is used to store computer execution instructions; and the processor 701 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0110] Alternatively, the memory 702 can be either standalone or integrated with the processor 701.

[0111] When the memory 702 is set up independently, the electronic device also includes a bus 703 for connecting the memory 702 and the processor 701.

[0112] This invention also provides a computer storage medium storing computer execution instructions, which, when executed by processor 701, implement the above method.

[0113] This invention also provides a computer program product, including a computer program that, when executed by a processor 701, implements the above-described method.

[0114] In the embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0115] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0116] Furthermore, the functional modules in the various embodiments of this invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit formed by the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0117] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor 701 to execute some steps of the methods of the various embodiments of this application.

[0118] It should be understood that the processor 701 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor, or the processor 701 can be any conventional processor 701. The steps of the method disclosed in this invention can be directly manifested as execution by the hardware processor 701, or execution by a combination of hardware and software modules within the processor 701.

[0119] The memory 702 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage device, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0120] Bus 703 can be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 703 can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus 703 in the accompanying drawings of this application is not limited to only one bus 703 or one type of bus 703.

[0121] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0122] An exemplary storage medium is coupled to a processor 701, enabling the processor 701 to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor 701. The processor 701 and the storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor 701 and the storage medium can exist as discrete components in an electronic device or a host device.

[0123] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A few-shot fine-tuning method for a pre-trained visual language model, characterized in that, Includes the following steps: Obtain several images, text, and the corresponding ground truth labels for each image, and construct a fine-tuning dataset; A pre-trained visual language model, a feature alignment module, and a local caching module are constructed. The pre-trained visual language model includes an image encoder, a text encoder, and a feature aggregation module. The image encoder and the text encoder are respectively connected to the feature aggregation module. Each image and text in the fine-tuning dataset is input into the pre-trained visual language model. The image encoder divides each image into several non-overlapping image blocks. Global features of the image and local features of each image block are extracted to form a feature set for each image. This feature set is then input into the feature alignment module to obtain the transformed feature set for each image. A storage matrix is ​​constructed in the local caching module and randomly initialized to obtain an initial storage matrix. The transformed feature sets of each image are iterated through. The initial storage matrix is ​​updated based on the transformed feature set of the first image to obtain the first updated storage matrix. The previously updated storage matrix is ​​then updated based on the transformed feature set of the current image to obtain the current updated storage matrix. Specifically, this includes: Calculate the cosine similarity between each feature in the transformed feature set of the current image and each cached entry in the previously updated storage matrix, and obtain the probability using the softmax function, as shown in the following formula: Among them, D i,j This represents the probability of assigning the i-th feature from the transformed feature set of the current image to the j-th cache entry in the previously updated storage matrix A. exp(·) represents the exponential function, and cos(·,·) represents calculating the cosine similarity between two vectors. i Let A represent the i-th feature in the transformed feature set of the current image. j This represents the j-th cache entry in the storage matrix A after the last update, where j = 1, 2, ..., M, and M represents the total number of cache entries; Based on the probability, each feature in the transformed feature set of the current image is assigned to the cache entry with the highest probability in the previously updated storage matrix A, as shown in the following formula: G j ={i|argmax k D i,k =j}; Among them, G j argmax represents the set of indices of all features in the transformed feature set of the current image that are assigned to the position of the j-th cache entry in the previously updated storage matrix A. k D i,k Indicates taking D i,k The maximum value of k in the range; Information is collected from each feature of the j-th cache entry assigned to the last updated storage matrix A, and the last updated storage matrix A is updated as follows: Among them, v h This represents one of the features assigned to the position of the j-th cache entry in the last updated storage matrix A, where γ represents the momentum coefficient, and A j Let A' represent the j-th cache entry in the previously updated storage matrix A. j This represents the j-th cached entry in the storage matrix after the current update. Each text is input into the text encoder to obtain a text embedding. The storage matrix after the current update and the text embedding are input into the feature aggregation module to obtain the current aggregated features. The predicted label corresponding to the current image is calculated based on the current aggregated features and the global features of the current image. A loss function is constructed using the predicted label and the true label. The pre-trained visual language model is fine-tuned based on the loss function to obtain a comparison between the pre-trained visual language model and the storage matrix after the last update.

2. The few-shot fine-tuning method for a pre-trained visual language model according to claim 1, characterized in that, The feature alignment module is a multilayer perceptron with two hidden layers. The feature set is transformed using this module to obtain the transformed feature set, as shown in the following equation: Where V represents the feature set, W1 represents the weights of the first hidden layer in the feature alignment module, W2 represents the weights of the second hidden layer in the feature alignment module, norm(·) represents the batch normalization operation, and σ(·) represents the ReLU activation function. Represents the transformed feature set. v i Let vi represent the i-th feature in the transformed feature set, where i represents the feature index. When i = 0, v0 represents the transformed global feature; when i = 1, 2, ..., N, vi ... N These represent the transformed local features of the 1st, 2nd to Nth image patches, respectively.

3. The few-shot fine-tuning method for a pre-trained visual language model according to claim 1, characterized in that, The updated storage matrix and the text embedding are input into the feature aggregation module to obtain the current aggregated features, specifically including: The text vector of the u-th category in the text embedding is matched with each cached entry in the updated storage matrix to obtain the matching weight, as shown in the following formula: Among them, E u W represents the text vector of the u-th category in the text embedding. u,j This represents the matching weight between the text vector of the u-th category in the text embedding and the j-th cache entry in the currently updated storage matrix; The detailed embedding of each category is calculated based on the matching weights and each cached entry in the current updated storage matrix, as shown in the following formula: in, This represents the detailed embedding of the current u-th category; The detailed description embedding of each current category is concatenated with the text vector of each category in the text embedding, and then aggregated using a linear layer and residual connections to obtain the aggregated features of each current category, as shown in the following formula: Where Linear(·) represents the function corresponding to the linear layer, and α represents the hyperparameter. This represents the aggregated feature of the current u-th category. The aggregated features of all categories constitute the current aggregated feature.

4. The few-shot fine-tuning method for a pre-trained visual language model according to claim 3, characterized in that, The predicted label for the current image is calculated based on the current aggregated features and the global features of the current image, specifically including: Calculate the similarity between the current aggregated features and the global features of the current image, and determine the predicted label based on the maximum likelihood, as shown in the following formula: Where y is the predicted label, and v0 represents the transformed global feature. This represents the aggregation feature of the current c-th category, where C is the total number of categories, c = 1, 2, ..., C, and τ represents the temperature coefficient.

5. A few-shot fine-tuning device for a pre-trained visual language model, characterized in that, include: The data acquisition module is configured to acquire several images, text, and the corresponding real labels for each image, and to construct a fine-tuning dataset; The model building module is configured to build a pre-trained visual language model, a feature alignment module, and a local caching module. The pre-trained visual language model includes an image encoder, a text encoder, and a feature aggregation module, wherein the image encoder and the text encoder are respectively connected to the feature aggregation module. The fine-tuning module is configured to input each image and text in the fine-tuning dataset into the pre-trained visual language model. The image encoder divides each image into several non-overlapping image blocks, extracts global features of the image and local features of each image block to form a feature set for each image, and inputs the feature set of each image into the feature alignment module to obtain the transformed feature set of each image. A storage matrix is ​​constructed in the local caching module, and the storage matrix is ​​randomly initialized to obtain an initial storage matrix. The transformed feature sets of each image are traversed, and the initial storage matrix is ​​updated based on the transformed feature set of the first image to obtain the first updated storage matrix. The previously updated storage matrix is ​​updated based on the transformed feature set of the current image to obtain the current updated storage matrix. Specifically, this includes: Calculate the cosine similarity between each feature in the transformed feature set of the current image and each cached entry in the previously updated storage matrix, and obtain the probability using the softmax function, as shown in the following formula: Among them, D i,j This represents the probability of assigning the i-th feature from the transformed feature set of the current image to the j-th cache entry in the previously updated storage matrix A. exp(·) represents the exponential function, and cos(·,·) represents calculating the cosine similarity between two vectors. i Let A represent the i-th feature in the transformed feature set of the current image. j This represents the j-th cache entry in the storage matrix A after the last update, where j = 1, 2, ..., M, and M represents the total number of cache entries; Based on the probability, each feature in the transformed feature set of the current image is assigned to the cache entry with the highest probability in the previously updated storage matrix A, as shown in the following formula: G j ={i|argmax k D i,k =j}; Among them, G j argmax represents the set of indices of all features in the transformed feature set of the current image that are assigned to the position of the j-th cache entry in the previously updated storage matrix A. k D i,k Indicates taking D i,k The maximum value of k in the range; Information is collected from each feature of the j-th cache entry assigned to the last updated storage matrix A, and the last updated storage matrix A is updated as follows: Among them, v h This represents one of the features assigned to the position of the j-th cache entry in the last updated storage matrix A, where γ represents the momentum coefficient, and A j Let A' represent the j-th cache entry in the previously updated storage matrix A. j This represents the j-th cached entry in the storage matrix after the current update. Each text is input into the text encoder to obtain a text embedding. The storage matrix after the current update and the text embedding are input into the feature aggregation module to obtain the current aggregated features. The predicted label corresponding to the current image is calculated based on the current aggregated features and the global features of the current image. A loss function is constructed using the predicted label and the true label. The pre-trained visual language model is fine-tuned based on the loss function to obtain a comparison between the pre-trained visual language model and the storage matrix after the last update.

6. A method for predicting based on image and text comparison, characterized in that, The method for fine-tuning the pre-trained visual language model using any one of claims 1-4, comparing the pre-trained visual language model with the last updated storage matrix, includes the following steps: Obtain the image to be processed and the input text; The image to be processed and the input text are input into the contrast-pre-trained visual language model. The image encoder divides the image to be processed into several non-overlapping image blocks, and extracts the global features of the image to be processed from each block. The input text is input into the text encoder to obtain the text vector corresponding to the input text. The last updated storage matrix and the text vector corresponding to the input text are input into the feature aggregation module to obtain aggregated features. The predicted label corresponding to the image to be processed is calculated based on the aggregated features and the global features of the image to be processed.

7. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-4.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-4.

Images