A detection method for a visual language large model with a fusion module

Abstract

The present application relates to a kind of detection methods for visual language large model with fusion module, comprising: the information to be measured including text and image is input into trained visual language large model, obtains target detection result graph;The fusion layer of each layer coding structure in the visual language large model includes: bimodal feature splicing module, fusion module, index slice module;The bimodal feature splicing module of i layer is used to obtain text splicing feature and image splicing feature i layer fusion module is used to in cross attention space, and fusion processing is carried out to and feature alignment output and i layer index slice module is used to respectively carry out index slice operation to and, output to the double modal feature to be processed of i+1 layer coding structure.The migration potential of visual language large model based on fusion module can be better played in the above-mentioned method, reduce the computing overhead, improve the calculation speed.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and in particular relates to a detection method for large visual language models with fusion modules. Background Technology

[0002] Recently, the rapid development of large-scale AI visual language models has greatly promoted the advancement of computer vision tasks. Large-scale visual language models typically have two branches: text and image encoders. However, large-scale visual language models with fusion modules are a type of visual-language pre-trained model that focuses more on object-level visual tasks, such as GLIP and FIBER. The characteristic of this type of large-scale visual language model with fusion modules is that it performs evidence-based pre-training on both object detection data and phrase localization data, and employs cross-attention structures to fuse the text and image dual-branch encoders at multiple levels. These cross-attention structures are the fusion modules.

[0003] To ensure the performance of large-scale visual-language models with fusion modules in downstream tasks, it is necessary to better transfer their general knowledge to various downstream tasks. The most direct strategy for transferring large-scale visual-language models to downstream tasks is to perform full fine-tuning of the pre-trained visual-language model on a specific task using a small number of samples or under full supervision. However, this strategy incurs enormous computational and storage resource overhead for increasingly large-scale visual-language models, while the transfer effect of fine-tuning only a portion of the structure is unsatisfactory. To address this issue, an efficient parameter fine-tuning method has been proposed, which sets only a small number of parameters as learnable and fine-tunes them to improve efficiency.

[0004] Currently, effective and efficient fine-tuning methods can be broadly categorized into two types: token-related methods and network-related methods. The former adds learnable tokens to the network input tokens for fine-tuning, hoping these learnable tokens can integrate knowledge from downstream tasks, such as CoOp (contextual optimization) and VPT (visual cue fine-tuning). The latter integrates lightweight network structures at specific locations in the vision-language pre-trained model, fine-tuning only these lightweight structures to achieve efficient transfer, such as adapters and adapterformers. However, regardless of the category, most current efficient fine-tuning methods only consider feature fine-tuning of a single encoder. For example, CoOp only adds learnable prefixes to text tokens for fine-tuning, or adapterformers only add residual bottleneck structures to the MLP (Multilayer Perceptron) layer of the image encoder to fine-tune visual features. However, these efficient fine-tuning methods are designed for a single branch, and for large vision-language models trained on both modalities, they cannot effectively supplement information from the other modality during downstream transfer. Dual-Modality Prompt Tuning (DPT) adds learnable tokens to both branches, resulting in better classification transfer performance.

[0005] However, the unique feature of large-scale visual language models with fusion modules lies in the fact that their powerful object retrieval capabilities largely stem from the multi-layered fusion modules. This structure highly couples the output features of the text and image encoders during pre-training. The parameter-efficient fine-tuning methods mentioned above, designed for single branches, cannot fully realize the potential of large-scale visual language models with fusion modules. Simply adding the previous parameter-efficient fine-tuning methods for single branches to the text and image encoders separately, the high coupling of the multi-layered fusion becomes an obstacle to optimization. This makes it difficult to select optimal hyperparameters for the dual-branch fine-tuning module and inevitably increases the number of parameters that need to be trained. Summary of the Invention

[0006] (a) Technical problems to be solved

[0007] To address the aforementioned problems in the prior art, this invention provides a detection method for large visual language models with fusion modules.

[0008] (II) Technical Solution

[0009] To achieve the above objectives, the main technical solutions adopted by the present invention include:

[0010] This invention provides a detection method for large visual language models with fusion modules, comprising:

[0011] Input the test information, including text and images, into the trained visual language model to obtain the object detection result image;

[0012] The trained visual language large model includes an L-layer coding structure. Each coding layer includes a bimodal encoder and a fusion layer. The fusion layer includes a bimodal feature concatenation module, a fusion module, and an index slicing module. L is a natural number greater than 5.

[0013] The processing of the i-th layer coding structure includes: the i-th layer bimodal encoder outputting text features P respectively. i and image features R i ;

[0014] The i-th layer's bimodal feature concatenation module is used to combine the text features to be concatenated obtained during the training phase with P. i By concatenating the text, we can obtain the text concatenation features. And the features of the images to be stitched obtained during the training phase are combined with R i By stitching together the images, we can obtain image stitching features.

[0015] The fusion module of layer i is used to perform [further processing] in the cross-attention space. and fusion processing and feature alignment output and

[0016] The index slicing module of the i-th layer is used to slice the index slices respectively. and Addition and After addition, perform index slicing operations on each, and output the result to be processed by the (i+1)th layer encoding structure. and Dimensions and P i The same dimensions Dimensions and R i They have the same dimensions.

[0017] Optionally, before inputting the test information, including text and images, into the trained visual language large model, the method further includes:

[0018] For the large visual language model to be trained, construct parameter fine-tuning components for the text features and image features to be concatenated required for each layer of the coding structure in the usage stage.

[0019] For each layer of the encoding structure, the parameter fine-tuning component is set in the fusion module of the encoding structure fusion layer. The parameter fine-tuning component includes: a meta-word vector component prototype token and a pseudo-inverse operation component corresponding to each modality; the initial parameter of the meta-word vector component is a random weight parameter.

[0020] Based on the training dataset, a large visual language model including the parameter fine-tuning components is trained. During the training process, the parameters of the modules of non-meta-word vector components in each layer are fixed parameters. After the large visual language model training converges, the training weight parameters of the meta-word vector components in each layer of the encoding structure are obtained. In each layer of the encoding structure, the meta-word vector components using the training weight parameters of that layer are inversely mapped to the pseudo-inverse operation components of each modality of that layer, respectively, to obtain the text features and image features to be concatenated required by the encoding structure of that layer in the usage stage.

[0021] Optionally, in each training process, the meta-word vector component of the current stage and the pseudo-inverse operation component of each modality are used to perform inverse mapping in each layer of the coding structure to obtain the text features to be concatenated and the image features to be concatenated used by the current layer of the coding structure in the current stage.

[0022] Based on the concatenated text features of each coding structure at the current stage, the features of the image to be concatenated and the features output by the bimodal encoder in that coding structure are concatenated and forward propagation and gradient backpropagation are performed. During training, the weight parameters of the meta-word vector components are updated until the visual language large model to be trained converges, and the training weight parameters of the meta-word vector components in each coding structure are obtained.

[0023] Optionally, n and m are the number of text tokens and image tokens, respectively, and d T and d I Z represents the dimensions of the text and image latent spaces, where T represents the text modality and I represents the image modality; i Z represents the original word vector components with training parameters, i.e., k learnable vectors. i ∈R k×d ;

[0024] The dual-modal feature concatenation module of the i-th layer is specifically used to obtain... and

[0025] Z i,(T) For Z i The text features to be concatenated are obtained by synchronous inverse mapping back to the text space. Z i,(I) For Z i The stitched image features obtained by synchronous inverse mapping back to image space

[0026] The fusion module of layer i is specifically used for:

[0027]

[0028] Among them, X-MHA i This represents the fusion process of the i-th layer;

[0029] Correspondingly, the (i+1)th layer bimodal encoder outputs the encoded text features P. i+1 and image features R i+1 ;

[0030]

[0031] This represents the text encoding process of a text encoder. This represents the image encoding process of an image encoder.

[0032] Optionally, the X-MHA i The fusion process includes:

[0033]

[0034]

[0035]

[0036] Among them, {W (q,mod) :mod∈{T,I}} is the query transformation matrix, and d is the dimension of the cross-attention space;

[0037] This is represented as the concatenated text query feature of the i-th layer. Let W represent the image query features concatenated at layer i, and Attn represent the attention weight matrix. i,(v,mod) :mod∈{T,I}} represents the value transformation matrix of the i-th layer, {W i,(out,mod) :mod∈{T,I}} represents the output transformation matrix of the i-th layer;

[0038] This represents the image-to-text fusion features and text-to-image fusion features output by the i-th layer fusion module.

[0039] Optionally, in each layer of the coding structure, the pseudo-inverse operation component corresponding to each modality includes: a text query inverse transformation matrix with inverse spatial transformation that does not require training. Image query inverse transformation matrix

[0040]

[0041] Features of the text to be concatenated

[0042] Features of the image to be stitched

[0043] Pinv(.) is the pseudo-inverse operation, B i,(q,omd) :mod∈{T,I}} is the bias vector of the query transformation in the i-th layer of the training process.

[0044] For the text features Z to be concatenated in the i-th layer i,(T) and the features Z of the image to be stitched i,(I) , and respectively with and They are in the same potential space.

[0045] Secondly, embodiments of the present invention also provide an efficient fine-tuning method for a large visual language model with a fusion module, comprising:

[0046] For the large visual language model to be trained, construct parameter fine-tuning components for the text features and image features to be concatenated required for each layer of the coding structure in the usage stage.

[0047] For each layer of the encoding structure, the parameter fine-tuning component is set in the fusion module of the encoding structure fusion layer. The parameter fine-tuning component includes: a meta-word vector component prototype token and a pseudo-inverse operation component corresponding to each modality; the initial parameter of the meta-word vector component is a random weight parameter.

[0048] Based on the training dataset, a large visual language model including the parameter fine-tuning components is trained. During training, the parameters of the modules of non-meta-word vector components in each layer are fixed. After the large visual language model training converges, the training weight parameters of the meta-word vector components in each layer of the encoding structure are obtained. In each layer of the encoding structure, the meta-word vector components with the training weight parameters of that layer are inversely mapped to the pseudo-inverse operation components of each modality of that layer, respectively, to obtain the text features and image features to be concatenated required by the encoding structure of that layer in the usage stage. Then, in the usage stage, the trained large visual language model is used to process the test information including text and images to obtain the target detection result map.

[0049] Optionally, in each training process, the meta-word vector component of the current stage and the pseudo-inverse operation component of each modality are used to perform inverse mapping in each layer of the coding structure to obtain the text features and image features to be concatenated used by the current layer of the coding structure in the current stage.

[0050] Based on the concatenated text features, image features to be concatenated, and features output by the bimodal encoder in each layer of the coding structure at the current stage, and performing forward propagation and gradient backpropagation, the weight parameters of the meta-word vector components are updated during training until the visual language large model to be trained converges, thus obtaining the training weight parameters of the meta-word vector components in each layer of the coding structure.

[0051] Optionally, the large visual language model to be trained is based on the GLIP network architecture, including a coding structure of more than L layers;

[0052] In each training session, the detection result y' = Head(P) is obtained. L ,R L The loss function is calculated by comparing y' with the true value y. If convergence is not achieved, the loss function is applied to the meta-word vector component Z in each layer of the coding structure. i The weight parameters are updated, and the bimodal features of the new training data are simultaneously incorporated into the training. L is a natural number greater than 5.

[0053] Thirdly, embodiments of the present invention provide a computing device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program in the memory and performs the steps of any of the methods described in the first aspect.

[0054] (III) Beneficial Effects

[0055] In the detection method of this invention, the fusion layer of the trained visual language large model can better fuse the text features and image features of the test information without increasing the amount of computation, ensuring the accuracy of the fusion result and improving the accuracy of the detection result.

[0056] The efficient fine-tuning method of this invention, based on SDPT, can better leverage the transfer potential of large-scale visual language models based on fusion modules, while using fewer parameters and reducing computational overhead. During the specific training process, the weight parameters of the original large-scale visual language model architecture are not changed. Instead, the fusion layer of each encoding structure is processed, and a meta-word vector component (Prototype Token) is added, thereby simplifying the training process, reducing computational overhead during training, ensuring rapid convergence, and improving training efficiency.

[0057] In this embodiment of the invention, SDPT uses the Prototype Token component to synchronously fuse bimodal features, i.e., bimodal knowledge. This requires minimal training parameters while achieving superior transfer performance, demonstrating excellent and efficient fine-tuning performance for large visual-language models based on the fusion module. Specifically, when k=10, SDPT requires even fewer training parameters, achieving significant fine-tuning results. Attached Figure Description

[0058] Figure 1 This is a schematic diagram illustrating the processing procedure of an existing large-scale visual language model.

[0059] Figure 2 This is a schematic diagram of the pre-training structure in the method of this invention and in existing methods;

[0060] Figure 3 This is a partial schematic diagram of the coding structure without the addition of a fusion module;

[0061] Figure 4 This is a partial schematic diagram of the encoding structure in the method of this embodiment of the invention;

[0062] Figure 5 This is a flowchart illustrating the pre-training process of the method in an embodiment of the present invention;

[0063] Figure 6 This is a schematic diagram of the target detection results of the method in an embodiment of the present invention. Detailed Implementation

[0064] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0065] Existing efficient fine-tuning methods for large-scale visual language models involve processing the text and image encoders on a single branch. While bimodal cue tuning (DPT) based on the importance of bimodal information supplementation exists, the learnable tokens for these two branches are asynchronous. This necessitates adding new cross-attention mechanisms to fuse the information from both branches, resulting in unnecessary storage and computational overhead for large-scale visual language models with fusion modules. Figure 1 As shown, it has a large computational cost, and requires training of all model parameters / weights during the training process, which increases the number of training parameters and thus the computational cost.

[0066] The purpose of this invention is to simultaneously fine-tune the parameters of a large visual language model with a fusion module using a single component on a dual-branch encoder to supplement bimodal information. This achieves better fine-tuning results and requires fewer model parameters to train compared to other efficient fine-tuning methods. The proposed method in this invention is called Synchronous Dual Prompt Tuning (SDPT).

[0067] Example 1

[0068] like Figure 1 As shown, this invention provides a detection method for a large visual language model with a fusion module. The method is executed by any computing device and includes the following steps:

[0069] The test information, including text and images, is input into the trained visual language model to obtain the object detection result image, such as... Figure 6 As shown.

[0070] In this embodiment, the trained visual language large model includes an L-layer coding structure, each layer of which includes a bimodal encoder and a fusion layer; the fusion layer includes a bimodal feature splicing module, a fusion module, and an index slicing module; L can be a natural number greater than 5, such as preferably L = 20.

[0071] The processing of the i-th layer encoding structure in the visual language large model includes: the i-th layer bimodal encoder outputting text features P respectively. i and image features R i ;

[0072] The i-th layer's bimodal feature concatenation module is used to combine the text features to be concatenated obtained during the training phase with P. i By concatenating the text, we can obtain the text concatenation features. And the features of the images to be stitched obtained during the training phase are combined with R i By stitching together the images, we can obtain image stitching features.

[0073] The fusion module of layer i is used to perform [further processing] in the cross-attention space. and fusion processing and feature alignment output and This represents the image-to-text fusion features and text-to-image fusion features output by the i-th layer fusion module;

[0074] The index slicing module of the i-th layer is used to slice the index slices respectively. and Addition and After addition, perform index slicing operations on each, and output the result to be processed by the (i+1)th layer encoding structure. and Dimensions and P i The same dimensions Dimensions and R i They have the same dimensions.

[0075] in, n and m are the number of text tokens and image tokens, respectively, and d T and d I Let T represent the dimensions of the text and image latent spaces, and I represent the text modality and I represent the image modality.

[0076] The dual-modal feature concatenation module of the i-th layer is specifically used to obtain... and

[0077] Z i,(T) For prototype token Z i The text features to be concatenated are obtained by synchronous inverse mapping back to the text space. Z i,(I) For Z i The stitched image features obtained by synchronous inverse mapping back to image space Z i Z represents the original word vector components with training parameters, i.e., k learnable vectors. i ∈R k×d For example, when k is 10, the number of training parameters is minimized, and the computer overhead is also minimized.

[0078] The fusion module of layer i is specifically used for: Among them, X-MHA i This represents the fusion process of the i-th layer.

[0079] The X-MHA i The fusion process includes:

[0080]

[0081]

[0082]

[0083] Among them, {W (q,mod) :mod∈{T,I}} is the query transformation matrix, and d is the dimension of the cross-attention space; This is represented as the concatenated text query feature of the i-th layer. Let W represent the image query features concatenated at layer i, and Attn represent the attention weight matrix. i,(v,mod) :mod∈{T,I}} represents the value transformation matrix of the i-th layer, {W i,(out,mod) :mod∈{T,I}} represents the output transformation matrix of the i-th layer;

[0084] This represents the image-to-text fusion features and text-to-image fusion features output by the i-th layer fusion module.

[0085] Correspondingly, the (i+1)th layer bimodal encoder outputs text features P. i+1 and image features R i+1 ;

[0086]

[0087] This represents the text encoding process of a text encoder. This represents the image encoding process of an image encoder.

[0088] In the detection method of this embodiment, the fusion layer of the trained visual language large model can better fuse the text features and image features of the test information without increasing the amount of computation, ensuring the accuracy of the fusion result and improving the accuracy of the detection result.

[0089] Of course, before using the trained visual language big model, the method also includes a pre-training phase of the visual language big model (i.e., the phase of training the visual language big model).

[0090] For the large visual language model to be trained, construct parameter fine-tuning components for the text features and image features to be concatenated required for each layer of the coding structure in the usage stage.

[0091] For each coding structure layer, the parameter fine-tuning component is set in the fusion module of the coding structure fusion layer. The parameter fine-tuning component includes: a prototype token component and a pseudo-inverse operation component corresponding to each modality; the initial parameter of the prototype token component is a random weight parameter.

[0092] Based on the training dataset, a large visual language model including the parameter fine-tuning components is trained. During training, the parameters of the modules of non-meta-word vector components in each layer of the encoding structure are fixed. After training convergence, the training weight parameters of the meta-word vector components in each layer of the encoding structure are obtained. In each layer of the encoding structure, the meta-word vector components using the training parameters are inversely mapped to the pseudo-inverse operation components of each modality to obtain the text features and image features to be concatenated required by the encoding structure of that layer in the usage stage.

[0093] In each training process, the meta-word vector component of the current stage and the pseudo-inverse operation component of each modality are used for inverse mapping to obtain the text features and image features to be concatenated used by the current layer coding structure in the current stage.

[0094] Based on the concatenated text features of each coding structure at the current stage, the features of the image to be concatenated and the features output by the bimodal encoder in that coding structure are concatenated and forward propagation and gradient backpropagation are performed. During training, the weight parameters of the meta-word vector components are updated until the visual language large model to be trained converges, and the training weight parameters of the meta-word vector components in each coding structure are obtained.

[0095] In each layer of the encoding structure, the pseudo-inverse operation component corresponding to each modality includes: a text query inverse transformation matrix that requires no training for inverse spatial transformation. Image query inverse transformation matrix

[0096]

[0097] Features of the text to be concatenated

[0098] Features of the image to be stitched

[0099] Pinv(.) is the pseudo-inverse operation, B (q,mod) :mod∈{T,I}} is the bias vector of the query transformation in the i-th layer of the training process.

[0100] For the text features Z to be concatenated in the i-th layer i,(T) and the features Z of the image to be stitched i,(I) , and respectively with and They are in the same potential space.

[0101] For example, the large visual language model to be trained is based on the GLIP network architecture and can be an encoding structure with L=20 layers.

[0102] In each training session, the detection result y' = Head(P) is obtained. L ,R L The loss function is calculated by comparing y' with the true value y. If convergence is not achieved, the weight parameters of the meta-word vector component Z in the large visual language model are adjusted. i The weight parameters are updated, and the bimodal features of the new task, i.e. the training data, are simultaneously incorporated into the training.

[0103] In this embodiment, the Prototype Token component is used to simultaneously fuse bimodal features, i.e., bimodal knowledge. This requires minimal training of weight parameters while achieving superior transfer performance, demonstrating excellent and efficient fine-tuning performance for large visual-language models based on the fusion module. Even when k=10, SDPT requires even fewer training weight parameters but still achieves significant fine-tuning results.

[0104] Example 2

[0105] To better understand the solutions of this embodiment, the explanation is based on the model framework and training content of existing methods, namely, a brief comparison between the SDPT of this embodiment and existing methods. Figure 2 As shown. Figure 2 (a) shows the pre-training process of an existing CoOp. Figure 2 (a) The structures marked with slashes are the structures that need to be trained; Figure 2 (b) shows the existing pre-training process of the Adaptformer. Figure 2 (b) The structures marked with slashes are the structures that need to be trained; Figure 2 (c) shows the existing DPT pre-training process. Figure 2 (c) The structures marked with slashes are the structures that need to be trained; Figure 2 (d) describes the pre-training process of SDPT in this embodiment. Figure 2 (d) The structures marked with slashes are the structures that need to be trained.

[0106] The fusion module structure of the visual language large-scale model based on the fusion module in this embodiment constructs a fusion module using a cross-attention space at each layer, and performs alignment (i.e., alignment of text features and image features) at different levels of the bimodal encoder. The image encoder of the visual language large-scale model based on the fusion module encodes each object in the image, generating visual embeddings of different regions to align one-to-one with the object words appearing in the text.

[0107] Let the text features (text token) and image features (image token) be denoted as follows: and Where R is a real number in mathematical terms, n and m are the number of text tokens and image tokens, respectively, and d T and d I Let T represent the dimensions of the latent space for text and images, with uppercase letters T and I representing the text modality and image modality, respectively.

[0108] In each layer of the large visual language model upon which the method of this embodiment is based, a text encoder layer is used. and image encoder layer P and R are encoded layer by layer. Assuming the total number of encoder layers in the large visual-language model based on the fusion module is L, the multi-layer fusion process of the large visual-language model based on the fusion module can be represented as:

[0109]

[0110]

[0111] P 0 R represents the text token output from a text encoder (such as the BERT model). 0 This represents the image token output from an image encoder such as the swintransformer network. and Let P represent the image-to-text fusion features and the text-to-image fusion features, respectively. All of these fusion features are P. i ,R i After X-MHA i The output.

[0112] X-MHA i The fusion process of the existing fusion module, which represents the bimodal cross-attention module of the i-th layer, is as follows (the following is an explanation of the layer number notation i omitted):

[0113]

[0114] P (v) =PW (v,T) ,R T2I =SoftMax(Attn)P (v) W (out,I) ,

[0115] R (v) =RW (v,I) ,P I2T =SoftMax(Attn) T )R (v) W (out,T) ,

[0116] Where {W (q,mod):mod∈{T,I}} is the query transformation matrix. After pre-training, P and R are both mapped from their own modal spaces to a common cross-attention space with a latent dimension of d, becoming the text query feature P. (q) and image query features R (q) .

[0117] Text query features P (q) and image query features R (q) Calculate the attention weight matrix Attn, which is the coupling between the two modalities, and finally the value transformation matrix W. (v,mod) and the output transformation matrix W (out,mod) To obtain the fusion features from one modality to another, i.e. and For simplicity, all bias terms of the transformations have been omitted, such as Figure 3 As shown.

[0118] This multi-layered fusion process enables the visual language big model based on the fusion module to achieve multi-layered coupled modeling in both text and image modalities, allowing it to output better object-level semantic features.

[0119] The SDPT in this embodiment is based on the fusion module of the visual language large model, and its core structure and method are as follows: Figure 4 As shown, it mainly includes Prototype Token (meta-word vector component) and inverse space transformation component that does not require training (i.e., pseudo-inverse operation component corresponding to each modality).

[0120] The Prototype Token is a single component that needs to be shared in both branches of the encoder layer. It synchronously supplements bimodal features, allowing the entire new task to be represented from both modalities simultaneously. It requires fewer parameters during training and effectively reduces computational overhead.

[0121] In the large-scale visual language model based on fusion modules, different network architectures are used to encode text and images separately, resulting in text tokens. and image token Naturally, they exist in different vector spaces. However, for the target categories of text and images in downstream tasks with practical significance, they must have achieved semantic alignment in some vector space, assuming that this vector space contains vectors that can fully express the task. In fact, the d-dimensional cross-attention space spanned in the fusion layer of the visual language large model based on the fusion module is precisely P. i and R iThis establishes a space for semantic alignment. Therefore, k learnable vectors, or Prototype Tokens, denoted as Z, are set in the i-th layer cross-attention space. i ∈R k ×d This ensures that the information from text and image modalities in downstream tasks is fully represented. Meanwhile, the inverse spatial transformation component, which requires no training, fully utilizes the weight parameters of the fusion module built upon the large visual-language model based on the fusion module, avoiding the introduction of additional training parameters.

[0122] During training, based on the current stage's Prototype Token and the inverse space transformation component that does not require training, such as... and Perform inverse mappings to obtain the text features to be concatenated used by the current layer's encoding structure at the current stage. and features of the image to be stitched The identifier of the i-th layer is omitted in the above parameter representation;

[0123] Based on the concatenated text features Z of each coding structure at the current stage (T) Features Z of the image to be stitched (I) The features P output by the dual-modal encoder in this layer's coding structure are respectively i R i The prototype tokens are concatenated and subjected to forward and backward propagation. During training, the weight parameters of the prototype tokens are updated until the large visual language model to be trained converges, thus obtaining the training weight parameters of the prototype tokens in each layer of the encoding structure, such as... Figure 4 As shown.

[0124] In this embodiment, the i-th layer is spliced ​​together. And together perform forward computation R:

[0125] The i-th layer fusion module performs a process on the i-th layer after splicing. and The splicing is represented as:

[0126]

[0127] Therefore, the encoding process of the (i+1)th layer dual-mode encoder is represented as:

[0128]

[0129] Among them, features with cap badges (e.g.) ) represents the concatenation of the original features and the token after inverse space transformation, and [k:k+n] and [k:k+m] represent the index slicing operation. and The bimodal features obtained by the bimodal encoder in the next layer are then used to output the prediction head of the final L layer, i.e., y' = Head(P L ,R L ).

[0130] During the pre-training phase, the detection result y' obtained through the prediction head layer is compared with the true value y to calculate the loss function, and Z is adjusted while freezing other network parameters. i The weights are updated, and the bimodal features of the new task are incorporated simultaneously. The aforementioned inverse space transformation component, which does not require training, makes full use of the weights of the fusion module built on the large visual-language model based on the fusion module, while avoiding the introduction of additional training weight parameters.

[0131] The pre-training described in this embodiment is the same as training, and they have the same meaning.

[0132] Combination Figure 5 As shown, the efficient fine-tuning method of this embodiment may include the following steps:

[0133] Step 1: Load the pre-trained, fusion-based visual-language model that needs to be transferred to downstream tasks (i.e., training data).

[0134] Step 2: Freeze all weights of the large visual language model.

[0135] Step 3: Based on the fusion module of the large visual language model, set prototype tokens in each layer of the fusion module. The number of tokens is k, and the specific value of k is determined by the user. The larger the value of k, the greater the training overhead. Set the tokens to be trainable. Preferably, k=10, which minimizes the training parameters.

[0136] Step 4: Based on the fusion module of the large visual-language model based on the fusion module, set up an inverse space transformation component that does not require training. and

[0137] Step 5: Update the parameters round by round using the loss function of the downstream task and the original loss function of the large visual-language model based on the fusion module. In each round, use the inverse space transformation component, which does not require training, to make the Prototype TokenZ... i The prototype token is synchronously inversely mapped back to the text space and image space, and then concatenated with the text token and image token respectively for forward propagation and gradient backpropagation to update the prototype token until the model converges. Figure 5 The fine-tuning shown can be understood as adjusting only the Prototype Token Z. iThe weights are assigned without adjusting other parameters and weight information in the model.

[0138] The training process refers to the process of using data and a loss function to update the parameters of a specific module in the model. During this process, the entire model needs to participate. "No need for training" means that the module does not need to update its parameters, i.e., it is frozen, but it still participates in the "training process." During use, all modules are frozen, meaning their parameters remain unchanged.

[0139] Step 6: Based on the model convergence, the Prototype Token Z of each fusion module in the large visual language model based on the fusion module. i The weight parameters are then combined with the inverse space transformation component. and Obtain the text features Z to be concatenated in the i-th layer. i(T) and the image features Z to be stitched i(I) So that during the usage phase, it can be directly used with the text features P output by the i-th layer bimodal encoder. i and image features R i Then, the parts are assembled.

[0140] During the usage phase, in the large visual language model based on the fusion module, the i-th layer encoding structure and the i-th layer dual-modal encoder each output text features P. i and image features R i The bimodal feature stitching module of the i-th layer is used to stitch Z... i(T) With P i By concatenating the text, we can obtain the text concatenation features. And Z i(I) With R i By stitching together the images, we can obtain image stitching features. The fusion module of layer i is used to perform [further processing] in the cross-attention space. and fusion processing and feature alignment output and The index slicing module of the i-th layer is used to respectively... and Perform index slicing operation and output the encoding structure to be processed at the (i+1)th layer. and Dimensions and P i The same dimensions Dimensions and R i They have the same dimensions.

[0141] Accordingly, the encoding of the (i+1)th layer bimodal encoder can be expressed as:

[0142] The target detection result is output by the prediction head / detection head after the Lth layer of coding structure.

[0143] The SDPT method in this embodiment better leverages the transfer potential of large visual-language models based on fusion modules compared to previous efficient fine-tuning methods, while using fewer parameters. Experimental results show that SDPT, using a single-component Prototype Token to simultaneously fuse bimodal knowledge, requires the fewest training parameters while achieving superior transfer performance, demonstrating excellent efficient fine-tuning performance for large visual-language models based on fusion modules. Even when k=10, SDPT requires fewer training parameters but still achieves significant fine-tuning results.

[0144] The finely tuned visual language model described above can be used for object detection. For example, using the GLIP network architecture, such as... Figure 1 As shown.

[0145] In this embodiment, at the start of the training phase, the GLIP model can first load pre-trained weights from the COCO natural large dataset. After loading, the model can be considered capable of detecting many common natural objects, but it will have difficulty detecting unfamiliar classes. Then, within the fusion module, the aforementioned Prototype Token and an inverse space transformation component that does not require training are inserted. The Prototype Token component has a small number of parameters, and its specific values ​​are randomly initialized. Next, the other parts of the network are frozen, and the weights of the Prototype Token are updated during training. The training samples are training datasets for various types of test information.

[0146] After training, during the testing phase, the GLIP network, which is loaded with COCO pre-trained weights and includes the trained Prototype Token component and inverse space transformation component, is used to perform object detection on the new class.

[0147] The above process makes the learned visual representations more suitable for object-level tasks, such as object detection, phrase localization, and region-conditional image generation.

[0148] Example 3

[0149] In addition, embodiments of the present invention also provide a computing device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program in the memory and performs the steps of any of the methods described in Embodiments 1 and 2 above.

[0150] This invention also provides a computer program product that corresponds to the steps described in any of the embodiments 1 and 2 above.

[0151] In the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0152] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make modifications, alterations, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A detection method for large visual language models with fusion modules, characterized in that, include: Input the test information, including text and images, into the trained visual language model to obtain the object detection result image; The trained visual language large model includes an L-layer coding structure. Each coding layer includes a bimodal encoder and a fusion layer. The fusion layer includes a bimodal feature concatenation module, a fusion module, and an index slicing module. L is a natural number greater than 5. The processing of the i-th layer coding structure includes: the i-th layer bimodal encoder outputting text features respectively. and image features ; The i-th layer's bimodal feature concatenation module is used to combine the features of the text to be concatenated obtained during the training phase with... By concatenating the text, we can obtain the text concatenation features. ; and combining the features of the images to be stitched obtained during the training phase with By stitching together the images, we can obtain image stitching features. ; The fusion module of layer i is used to perform [further processing] in the cross-attention space. and Fusion processing and feature alignment output and ; The index slicing module of the i-th layer is used to slice the index slices respectively. and Addition and After addition, perform index slicing operations on each, and output the result to be processed by the (i+1)th layer encoding structure. and ; Dimensions and The same dimensions Dimensions and The dimensions are the same; , , where n and m are the number of text tokens and image tokens, respectively. and Let T represent the dimensions of the text and image latent spaces, and I represent the text modality and I represent the image modality. This represents the original word vector components with training parameters, i.e., k learnable vectors. ; The dual-modal feature concatenation module of the i-th layer is specifically used to obtain... and ; for The text features to be concatenated are obtained by synchronous inverse mapping back to the text space. , for The stitched image features obtained by synchronous inverse mapping back to image space ; The fusion module of layer i is specifically used for: ； in, This represents the fusion process of the i-th layer; Correspondingly, the (i+1)th layer bimodal encoder outputs the encoded text features. and image features ; ； This represents the text encoding process of a text encoder. This represents the image encoding process of an image encoder; The The fusion process includes: ； ； ； in,{ } represents the query transformation matrix, and d represents the dimension of the cross-attention space; This is represented as the text query feature concatenated at the i-th layer. The query features of the image after concatenation at the i-th layer are represented as follows. Represented as an attention weight matrix, { } represents the value transformation matrix of the i-th layer, { } represents the output transformation matrix of the i-th layer; This represents the image-to-text fusion features and text-to-image fusion features output by the i-th layer fusion module; In each layer of the encoding structure, the pseudo-inverse operation component corresponding to each modality includes: a text query inverse transformation matrix that requires no training for inverse spatial transformation. Image query inverse transformation matrix ; ； Features of the text to be concatenated ; Features of the image to be stitched ; in, This is a pseudo-inverse operation. } represents the bias vector for the query transformation at the i-th layer during the training process. For the text features to be concatenated in the i-th layer and features of the image to be stitched , and respectively with and They are in the same potential space.

2. The method according to claim 1, characterized in that, Before inputting the test information, including text and images, into the trained large visual language model, the method further includes: For the large visual language model to be trained, construct parameter fine-tuning components for the text features and image features to be concatenated required for each layer of the coding structure in the usage stage. For each layer of the encoding structure, the parameter fine-tuning component is set in the fusion module of the encoding structure fusion layer. The parameter fine-tuning component includes: a meta-word vector component prototype token and a pseudo-inverse operation component corresponding to each modality; the initial parameter of the meta-word vector component is a random weight parameter. Based on the training dataset, a large visual language model including the parameter fine-tuning components is trained. During the training process, the parameters of the modules of non-meta-word vector components in each layer are fixed parameters. After the large visual language model training converges, the training weight parameters of the meta-word vector components in each layer of the encoding structure are obtained. In each layer of the encoding structure, the meta-word vector components using the training weight parameters of that layer are inversely mapped to the pseudo-inverse operation components of each modality of that layer, respectively, to obtain the text features and image features to be concatenated required by the encoding structure of that layer in the usage stage.

3. The method according to claim 2, characterized in that, In each training process, the meta-word vector component of the current stage and the pseudo-inverse operation component of each modality are used to perform inverse mapping in each layer of the coding structure to obtain the text features and image features to be spliced ​​used by the current layer of the coding structure in the current stage. Based on the concatenated text features, image features to be concatenated, and features output by the bimodal encoder in each layer of the coding structure at the current stage, and performing forward propagation and gradient backpropagation, the weight parameters of the meta-word vector components are updated during training until the visual language large model to be trained converges, thus obtaining the training weight parameters of the meta-word vector components in each layer of the coding structure.

4. An efficient fine-tuning method for large visual language models with fusion modules, characterized in that, include: For the large visual language model to be trained, construct parameter fine-tuning components for the text features and image features to be concatenated required for each layer of the coding structure in the usage stage. For each layer of the encoding structure, the parameter fine-tuning component is set in the fusion module of the encoding structure fusion layer. The parameter fine-tuning component includes: a meta-word vector component prototype token and a pseudo-inverse operation component corresponding to each modality; the initial parameter of the meta-word vector component is a random weight parameter. Based on the training dataset, a large visual language model including the parameter fine-tuning component is trained. During training, the parameters of the modules of non-meta-word vector components in each layer are fixed. After the large visual language model training converges, the training weight parameters of the meta-word vector components in each layer of the encoding structure are obtained. In each layer of the encoding structure, the meta-word vector components with the training weight parameters of that layer are inversely mapped to the pseudo-inverse operation components of each modality of that layer, respectively, to obtain the text features and image features to be concatenated required by the encoding structure of that layer in the usage stage. Then, in the usage stage, the trained large visual language model is used to process the test information including text and images, and the detection method described in claim 1 is used to obtain the target detection result image.

5. The method according to claim 4, characterized in that, In each training process, the meta-word vector component of the current stage and the pseudo-inverse operation component of each modality are used to perform inverse mapping in each layer of the coding structure to obtain the text features and image features to be spliced ​​used by the current layer of the coding structure in the current stage. Based on the concatenated text features, image features to be concatenated, and features output by the bimodal encoder in each layer of the coding structure at the current stage, and performing forward propagation and gradient backpropagation, the weight parameters of the meta-word vector components are updated during training until the visual language large model to be trained converges, thus obtaining the training weight parameters of the meta-word vector components in each layer of the coding structure.

6. The method according to claim 5, characterized in that, The large visual language model to be trained is based on the GLIP network architecture, including a coding structure of more than L layers. Obtain the test results in each training session. and will Compared with the true value The loss function is calculated. If convergence is not achieved, then the meta-word vector components in each layer of the coding structure are adjusted. The weight parameters are updated, and the bimodal features of the new training data are simultaneously incorporated into the training. L is a natural number greater than 5.

7. A computing device, characterized in that, The method includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program in the memory and performs the steps of the method according to any one of claims 1 to 6.

Images