Zero-sample cross-modal image retrieval method and device based on low-rank adaptation

Abstract

The invention discloses a zero-sample cross-modal image retrieval method and device based on low-rank adaptation. The method comprises the following steps: inputting a to-be-retrieved first modal image into a retrieval model to obtain an output vector; and comparing the output vectors with predetermined output vectors of various types of second modal image test samples, and screening out low-difference second modal image test samples to form a retrieval set. The retrieval model is obtained by replacing a corresponding layer in a transformer block of a large model DINOv2 with a domain-independent and domain-related module by using a preset low-rank technology so as to increase parallel low-rank matrix branches, and freezing original network parameter gradient training and verification; according to the method, for similar points and unique points of two different modalities, domain-independent and domain-related modules are designed to replace partial layers in DINOv2, so that the low-rank technology can better adapt to cross-modal zero sample image retrieval tasks, and the model robustness is improved. During training, original parameters are frozen, only the low-rank matrix is optimized, and better accuracy can be provided while training parameters are reduced.

Description

Technical Field

[0001] The present invention belongs to the field of image retrieval, and in particular relates to a zero-sample cross-modal image retrieval method and device based on low-rank adaptation. Background Art

[0002] Existing zero-shot image retrieval methods are mainly divided into two types: knowledge distillation-based methods and large model-based methods.

[0003] 1) Knowledge distillation-based methods (Liu et al. 2019; Tian et al. 2022) aim to enable the student model to retain the knowledge learned by the teacher model in the pre-training stage to prevent it from being forgotten in downstream tasks. These methods usually adopt a training method that fixes the gradient of the teacher model and transfers knowledge to the student model in a direction, which limits the generalization ability of the model to a certain extent. Liu et al. (2023) proposed a method for bidirectional knowledge alignment between teacher and student models. The teacher can dynamically fine-tune the knowledge passed to the student in the next stage based on the cross-modal knowledge learned by the student model to better adapt to the context of multimodal retrieval. However, the above methods require training two or more models at the same time, and only one of the models is used during inference, which has the problem of poor training efficiency.

[0004] 2) Methods based on large models (Sain et al. 2023; Koley et al. 2024). In recent years, various large language-vision models have emerged. It is expected to use the excellent performance of the contrastive language-image pre-trained large model (CLIP) in the field of multimodal learning to solve the image retrieval problem. The CLIP model is pre-trained on a very large-scale image-text dataset. Thanks to the huge amount of parameters and training data, it has rich feature representations in downstream tasks and demonstrates strong generalization capabilities, enabling the model to fully understand the local similarity semantics of the image to be retrieved and the image of another modality to be retrieved in the image retrieval task. Although large models perform well in various fields, existing methods for zero-shot image retrieval often directly use large models in order to utilize their rich feature representations, which requires a large amount of training data and hardware resources. At the same time, due to the large number of parameters, the model inference speed is slow. Limited by the huge number of parameters, it is difficult to be directly used for training in downstream tasks. Therefore, in the field of image retrieval, how to efficiently fine-tune these large models is becoming increasingly important. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a zero-sample cross-modal image retrieval method and device based on low-rank adaptation. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] In a first aspect, an embodiment of the present invention provides a zero-sample cross-modal image retrieval method based on low-rank adaptation, comprising:

[0007] Obtain a retrieval model; wherein, the retrieval model is obtained by replacing the corresponding layers in the transformer block of the large model DINOv2 with pre-designed domain-independent modules and domain-dependent modules by using a preset low-rank technology, thereby adding parallel low-rank matrix branches to the original weights, and freezing the original network parameter gradients, and then training with a training set containing first modality image training samples and second modality image training samples, and verifying with a test set after the training is completed; the first modality and the second modality are different modalities, and the categories of samples in the test set and the training set are different;

[0008] Inputting the first modality image to be retrieved into the retrieval model to obtain a corresponding output vector;

[0009] The output vector of the first modality image to be retrieved and the output vector of each category of the second modality image test samples determined in the test set verification process are compared respectively, and a preset number of second modality image test samples that meet the low difference requirement are determined to form a retrieval set.

[0010] In one embodiment of the present invention, the training process of the retrieval model includes:

[0011] Get the large model DINOv2 whose network parameters have been pre-trained;

[0012] Using the preset low-rank technology, the corresponding layers in some preset modules in the first N-1 transformer blocks of the large model DINOv2 are replaced with the domain-independent modules, the corresponding layers in some preset modules in the Nth transformer block are replaced with the domain-independent modules and the domain-related modules, and the corresponding layers in some preset modules in the Nth transformer block are replaced with the domain-related modules, so as to obtain a constructed retrieval model; wherein the domain-independent modules and the domain-related modules are obtained by retaining the original weights in the replaced layers and adding low-rank matrix branches for parallel addition, and the low-rank matrix branches are composed of low-rank matrices constructed by the preset low-rank technology connected in series;

[0013] Freeze the original network parameter gradient of the large model DINOv2, and for the current round of training, input a small batch of training sets containing first modality image training samples and second modality image training samples into the built retrieval model, and obtain the output vector of each training sample through feature extraction processing of the retrieval model; wherein the categories of the training samples in the training set are known;

[0014] For each first modality image training sample, use its output vector, the output vectors of the second modality image training samples of the same category and different categories to calculate the value of its loss function, and obtain the value of the total loss function of all the first modality image training samples in the small batch;

[0015] Based on the gradient of the total loss function, the low-rank matrices in each domain-independent module and domain-dependent module are optimized to complete the current round of training;

[0016] Repeat multiple rounds of training until the preset maximum number of training rounds is reached to obtain a pre-trained retrieval model.

[0017] In one embodiment of the present invention, when the preset low-rank technology is a low-rank adaptive LoRA technology, the preset low-rank technology is used to replace the corresponding layers in the partial preset modules in the first N-1 transformer blocks of the large model DINOv2 with the domain-independent modules, and the corresponding layers in the partial preset modules in the Nth transformer block are replaced with the domain-independent modules and the domain-related modules, and the corresponding layers in the partial preset modules in the Nth transformer block are replaced with the domain-related modules, to obtain a constructed retrieval model, including:

[0018] For the N transformer blocks of the large model DINOv2, the low-rank adaptive LoRA technology is used to replace the query-key-value layer, the projection layer in the multi-head attention module in the first N-1 transformer blocks, and the two linear layers in the linear module with the domain-independent module, the query-key-value layer in the multi-head attention module in the Nth transformer block is replaced with the domain-independent module, the projection layer is replaced with the domain-related module, and the two linear layers in the linear module in the Nth transformer block are replaced with the domain-related module, so as to obtain a completed retrieval model.

[0019] In one embodiment of the present invention, for the current round of training, a small batch of training sets containing first modality image training samples and second modality image training samples is input into the constructed retrieval model, and an output vector of each training sample is obtained through feature extraction processing of the retrieval model, including:

[0020] After the small batch of the current round of training is input into the retrieval model, each training sample is mapped into a vector with position information using the patch embedding layer at the input end of the large model DINOv2 to obtain the input vector of each training sample;

[0021] The input vector of each training sample is feature extracted in turn using N transformer blocks, and the output vector of each training sample is output through the multi-layer perceptron layer at the end of the large model DINOv2; wherein, the weight branch and the low-rank matrix branch in the domain-independent module simultaneously receive the vector data of the first modality image training sample and the second modality image training sample; in the domain-dependent module, the weight branch is respectively connected in parallel with the two low-rank matrix branches and added, and its weight branch simultaneously receives the vector data of the first modality image training sample and the second modality image training sample, one low-rank matrix branch receives the vector data of the first modality image training sample, and the other low-rank matrix branch receives the vector data of the second modality image training sample; the weight matrix W in all weight branches is frozen; each low-rank matrix branch includes a low-rank matrix B and a low-rank matrix A connected in series; the dimension of the weight matrix W is d×k; the dimensions of the low-rank matrix B and the low-rank matrix A are d×r and r×k respectively, r is the rank, and the value of r meets the low-rank requirement.

[0022] In one embodiment of the present invention, each training sample is mapped into a vector with position information using the patch embedding layer at the input end of the large model DINOv2 to obtain an input vector of each training sample, including:

[0023] The patch embedding layer is used to evenly divide the image corresponding to each input training sample into K blocks; and each block image is mapped into a D-dimensional vector;

[0024] Add a learnable classification vector x class , which is used as the output of the retrieval model, and the input vector is constructed for each training sample as follows:

[0025]

[0026] in, The image corresponding to the training sample is evenly divided into K blocks and mapped respectively. The vector after the j-th block image is mapped is D-dimensional; E pos Represents the relative position information of all elements in z0, in vector form.

[0027] In one embodiment of the present invention, for each first modality image training sample, the output vector and the output vectors of the second modality image training samples of the same category and different categories are used to calculate the value of the loss function, which is expressed as:

[0028]

[0029] in, is the first modality image training sample The output vector, L i is the first modality image training sample The value of the loss function, For The output vector of the second modality image training sample of the same category, For Output vectors of the second modality image training samples of different categories, d(,) is the calculated Euclidean distance, and α is the interval hyperparameter;

[0030] The value of the total loss function of all first modality image training samples of the small batch is expressed as follows:

[0031]

[0032] Where M is the number of all first modality image training samples in this small batch.

[0033] In one embodiment of the present invention, the low-rank matrices in each domain-independent module and domain-dependent module are optimized based on the gradient of the total loss function to complete the current round of training, which is implemented using the Adam optimizer.

[0034] In one embodiment of the present invention, the test set verification process includes:

[0035] Inputting the test set into the pre-trained retrieval model to obtain an output vector of each test sample; wherein the test set contains a first modality image test sample and a second modality image test sample, and the category is known;

[0036] For any first modality image test sample, the output vector thereof is compared with the output vectors of the second modality image test samples of each category, and a preset number of second modality image test samples that meet the low difference requirement are determined to form a neighboring set of the first modality image test sample; the number of second modality image test samples in the neighboring set that are consistent with the category of the first modality image test sample is determined, so as to obtain the accuracy of the retrieval model corresponding to the first modality image test sample;

[0037] The retrieval model accuracy corresponding to all first modality image test samples is used to obtain the final accuracy of the retrieval model, and verification is passed when the final accuracy meets the requirements.

[0038] In one embodiment of the present invention, for any first modality image test sample, the output vector thereof is compared with the output vectors of the second modality image test samples of each category, and a preset number of second modality image test samples that meet the low difference requirement are determined to form a neighboring set of the first modality image test sample, including:

[0039] For any first modality image test sample, the Euclidean distance is calculated for its output vector and the output vectors of the second modality image test samples of each category, and the samples are sorted from small to large according to the Euclidean distance, and a preset number of second modality image test samples ranked first are taken to form a neighboring set.

[0040] In a second aspect, an embodiment of the present invention provides a zero-sample cross-modal image retrieval device based on low-rank adaptation, comprising:

[0041] A retrieval model acquisition module is used to acquire a retrieval model; wherein, the retrieval model is obtained by replacing the corresponding layers in the transformer block of the large model DINOv2 with pre-designed domain-independent modules and domain-dependent modules by using a preset low-rank technology, thereby adding parallel low-rank matrix branches to the original weights, and freezing the original network parameter gradients, and then training with a training set containing first modality image training samples and second modality image training samples, and verifying with a test set after the training is completed; the first modality and the second modality are different modalities, and the categories of samples in the test set and the training set are different;

[0042] A retrieval model processing module, used for inputting the first modality image to be retrieved into the retrieval model to obtain a corresponding output vector;

[0043] The retrieval set determination module is used to compare the output vectors of the first modality image to be retrieved with the output vectors of the second modality image test samples of each category determined during the test set verification process, and determine a preset number of second modality image test samples that meet the low difference requirement to form a retrieval set.

[0044] Beneficial effects of the present invention:

[0045] The embodiment of the present invention uses low-rank technology instead of directly training a large model to explore the field of cross-modal image retrieval such as zero-sample sketch retrieval. The overall model is based on the Vision Transformer architecture of the large model DINOv2. Domain-independent modules and domain-dependent modules are designed for the similarities and unique points of two different modal images, and the corresponding layers in some modules of DINOv2 are replaced. Specifically, a low-rank matrix constructed by low-rank technology is added to the original weight matrix as a low-rank branch, so that the low-rank technology can better adapt to the zero-sample image retrieval task belonging to cross-modal retrieval, and increase the robustness of the model. The original parameters of DINOv2 are frozen during the training process, and only the low-rank matrices in the domain-independent module and the domain-dependent module are optimized. The present invention introduces large-model low-rank adaptation technology into cross-modal image retrieval tasks such as zero-sample sketch retrieval, which can reduce the model training parameters while maintaining or even better than the accuracy of training a complete large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of a process flow of a zero-sample cross-modal image retrieval method based on low-rank adaptation provided by an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of the process of training a retrieval model in an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of the structure of a retrieval model in an embodiment of the present invention;

[0049] Figure 4 A schematic diagram of the structure of a zero-sample cross-modal image retrieval device based on low-rank adaptation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0051] Among the existing zero-shot image retrieval methods, the knowledge distillation-based method cannot better adapt to the multimodal image retrieval task, and the training efficiency is not high. This method usually uses a pre-trained teacher model to transfer knowledge to the student model backbone. However, since the data used to train the teacher model often contains a relatively single modality, it cannot help the student model backbone learn the discriminative information of the image to be retrieved. In addition, in the model reasoning stage, the teacher model cannot participate in image retrieval, resulting in poor overall training efficiency of the model. The large model-based method is based on a large number of multimodal data sets during its pre-training, so it can better handle cross-modal image retrieval tasks and does not require the teacher model to provide additional semantic information. However, due to its huge number of parameters, the training process requires more hardware resources, the training cost is high, and there is also the problem of slow reasoning speed.

[0052] In order to overcome the above problems, the present invention is based on low-rank technology, combines the characteristics of two modal domain images, and designs a zero-sample cross-modal image retrieval method and device based on low-rank adaptation. Figure 1 , the method may include the following steps:

[0053] S1, obtain the retrieval model;

[0054] The retrieval model is obtained by replacing the corresponding layers in the transformer block of the large model DINOv2 with pre-designed domain-independent modules and domain-dependent modules by using a preset low-rank technology, thereby adding parallel low-rank matrix branches to the original weights, freezing the original network parameter gradients, and then training with a training set containing first modality image training samples and second modality image training samples, and verifying with a test set after the training; the first modality and the second modality are different modalities, and the categories of samples in the test set and the training set are different;

[0055] It should be noted that in S1, the retrieval model no longer freezes the original weight matrix when it is used. The model parameters at this time include the previously frozen parameters and all low-rank matrices determined in the training phase.

[0056] S2, inputting the first modality image to be retrieved into the retrieval model to obtain a corresponding output vector;

[0057] S3, comparing the output vector of the first modality image to be retrieved with the output vectors of the second modality image test samples of each category determined during the test set verification process, and determining a preset number of second modality image test samples that meet the low difference requirement to form a retrieval set.

[0058] The structure, processing process, output vector of the retrieval model and the process of performing difference comparison to obtain the retrieval set are described in detail later.

[0059] In the embodiment of the present invention, the first modality and the second modality are different modalities, which embodies cross-modal image retrieval. The images of the first modality and the second modality in the embodiment of the present invention may be of different modalities without any specific limitation. For example, in an optional implementation, the image of the first modality may be a sketch, and the image of the second modality may be a photo.

[0060] In the embodiment of the present invention, the sample categories of the training set and the test set are completely different, so that the knowledge learned by the model in the training stage is tested on completely new categories, which embodies the idea of ​​"zero sample".

[0061] The embodiment of the present invention uses the preset low-rank technology to improve the existing large model DINOv2, and replaces some corresponding layers in its transformer block with the domain-independent modules and domain-dependent modules pre-designed by the present invention. For the replaced layers, this replacement method is reflected in retaining their original weights and adding low-rank matrix branches for parallel connection constructed by low-rank technology, thereby adjusting the model structure. The domain-independent modules and domain-dependent modules are designed after considering the similarities and differences between the first modality image to be retrieved and the other modality image to be retrieved, namely the second modality image.

[0062] In order to facilitate understanding of the solution of the present invention, the training process of the retrieval model is first described. Figure 2 , the training process of the retrieval model includes the following steps:

[0063] S01, obtain the large model DINOv2 whose network parameters have been pre-trained;

[0064] The large model DINOv2 is an existing large model, whose structure includes a patch embedding layer (PatchEmbedding layer, referred to as Patch layer), N transformer blocks and a multi-layer perceptron layer (MLP head layer) connected in sequence.

[0065] The present invention can use the pre-training parameters of the already disclosed large model DINOv2, that is, obtain the pre-trained large model DINOv2.

[0066] S02, using the preset low-rank technology, replace the corresponding layers of some preset modules in the first N-1 transformer blocks of the large model DINOv2 with the domain-independent modules, replace the corresponding layers of some preset modules in the Nth transformer block with the domain-independent modules and the domain-dependent modules, and replace the corresponding layers of some preset modules in the Nth transformer block with the domain-dependent modules, to obtain a completed retrieval model;

[0067] The domain-independent module and the domain-dependent module are obtained by retaining the original weights in the replaced layer and adding low-rank matrix branches for parallel addition, and the low-rank matrix branches are formed by connecting low-rank matrices constructed by the preset low-rank technology in series;

[0068] In an embodiment of the present invention, the preset low-rank technology may be a low-rank adaptive LoRA technology, but is certainly not limited thereto.

[0069] In order to facilitate understanding of the solution of the present invention, the low-rank adaptation technology is briefly introduced below.

[0070] Low-rank adaptation technology (LoRA, Hu et al. 2021) is a lightweight training method that can significantly reduce the number of training parameters and training overhead, while reaching or even exceeding the accuracy of training a complete model. In specific implementation, LoRA inserts a small number of low-rank matrix weights on the basis of freezing the entire model gradient. The forward propagation of the conventional model can be described mathematically as follows:

[0071]

[0072] Among them, W i ∈R d×k , b i ∈R k are the original weights and biases of the i-th layer, respectively. Where d and k are natural numbers.

[0073] When LoRA fine-tunes a large model, it freezes the weights W of the original model. i and deviation b i Two low-rank matrices A and B are introduced to participate in forward propagation. The mathematical description is as follows:

[0074]

[0075] Among them, A i , B i are the low-rank matrices A and B of the i-th layer respectively; for low-rank matrices A and B, B∈R d×r , A∈R r×k , r<<min(d,k), indicating low rank. During inference, matrices A and B can be merged into W i In this way, we ensure that the introduction of low-rank matrices does not add any additional latency to the inference process.

[0076] Since the weights and biases are frozen, the model training and inference process only involves these low-rank matrices. Since the number of parameters of low-rank matrices is small, LoRA has the advantage of low training cost. Therefore, LoRA is an efficient method for fine-tuning large models.

[0077] On this basis, the present invention can introduce low-rank adaptive technology into the field of zero-sample image retrieval, and design retrieval frameworks for different domains, so that the technology can better adapt to the learning of two-modality images. For ease of understanding, the following text uses the low-rank adaptive LoRA technology, the image of the first modality is a sketch, and the image of the second modality is a photo, and zero-sample sketch retrieval is performed as a specific example, but it should be understood that this example does not constitute a limitation to the solution of the present invention.

[0078] Among them, when the preset low-rank technology is the low-rank adaptive LoRA technology, S02 may include:

[0079] For the N transformer blocks of the large model DINOv2, the low-rank adaptive LoRA technology is used to replace the query-key-value layer, the projection layer in the multi-head attention module in the first N-1 transformer blocks, and the two linear layers in the linear module with the domain-independent module, the query-key-value layer in the multi-head attention module in the Nth transformer block is replaced with the domain-independent module, the projection layer is replaced with the domain-related module, and the two linear layers in the linear module in the Nth transformer block are replaced with the domain-related module, so as to obtain a completed retrieval model.

[0080] When the LoRA technology is used in the embodiment of the present invention, the domain-independent module is recorded as a DA-LoRA module, and the domain-dependent module is recorded as a DS-LoRA module. The low-rank matrix branch is composed of low-rank matrices A and B constructed by the LoRA technology connected in series.

[0081] In the large model DINOv2, the multi-head attention module is the Attention module, the query-key-value layer is the QKV layer, the projection layer is the Projection layer, and the linear module is the MLP module. For details, please refer to the relevant technical understanding of the large model DINOv2, which will not be explained in detail here.

[0082] See also Figure 3 Understand the structure of the built retrieval model. The first N-1 transformer blocks are TransformerBlock (see Figure 3 The Nth transformer block is the Transformer Last Block (see Figure 3 For simplicity, only one of the first N-1 transformer blocks is shown as an example.

[0083] As a specific example, it can be seen that for the QKV layer, Projection layer, and MLP module ( Figure 3 The two linear layers in the multi-layer perceptron in the MLP (i.e., Linear1 and Linear2 in the MLP) are replaced with DA-LoRA modules. The structure of the DA-LoRA module is shown in the light blue part, which has two branches. The left one is the original weight W that is retained, which can be used as the original weight branch, and the right one is the low-rank matrix branch newly added in the embodiment of the present invention, including the low-rank matrices B and A connected in series. The low-rank matrix branch is constructed according to the LoRA technology. The two branches are connected in parallel, and the corresponding output results are summed.

[0084] For the Nth transformer block, the QKV layer in the Attention module is replaced with a DA-LoRA module, the Projection layer in the Attention module, and the two linear layers in the MLP module (i.e., Linear1 and Linear2 in MLP) are all replaced with DS-LoRA modules. The structure of the DS-LoRA module is shown in the yellow part, with three branches. The left side is the original weight W that is retained, which can be used as the original weight branch, and the two on the right are the low-rank matrix branches newly added in the embodiment of the present invention, both of which include low-rank matrices B and A connected in series, and the low-rank matrix branches are constructed according to the LoRA technology. Each low-rank matrix branch is connected in parallel with the original weight branch, and the corresponding output results are summed.

[0085] from Figure 3 It can be seen that each low-rank matrix branch includes a low-rank matrix B and a low-rank matrix A connected in series; however, for all DA-LoRA modules and DS-LoRA modules, the low-rank matrices B of different low-rank matrix branches may be different. Similarly, the low-rank matrices A of different low-rank matrix branches may be different.

[0086] The dimension of the weight matrix W is d×k; the dimensions of the low-rank matrix B and low-rank matrix A are d×r and r×k respectively, where r is the rank. The value of r satisfies the low-rank requirement.

[0087] In the embodiment of the present invention, the DA-LoRA module and the DS-LoRA module do not change the structure of the original module in the Vision Transformer (i.e., the large model DINOv2), but only add parallel low-rank matrix branches (including two low-rank matrices A and B) on the side. This design is independent of the specific structure of the model and has excellent portability and plug-and-play characteristics.

[0088] The usage of DA-LoRA module and DS-LoRA module will be explained later.

[0089] S03, freezing the original network parameter gradient of the large model DINOv2, and for the current round of training, inputting a small batch of training sets containing first modality image training samples and second modality image training samples into the built retrieval model, and obtaining an output vector of each training sample through feature extraction processing of the retrieval model;

[0090] Wherein, the categories of the training samples in the training set are known;

[0091] The original network parameter gradients of the large model DINOv2 are frozen, that is, the weight matrices W in all weight branches are frozen. The parameters to be learned are the low-rank matrices A and B in the DA-LoRA module and the DS-LoRA module.

[0092] For the current round of training, a small batch of training sets containing first modality image training samples and second modality image training samples is input into the constructed retrieval model, and an output vector of each training sample is obtained through feature extraction processing of the retrieval model, including:

[0093] 1) After the small batch of the current round of training is input into the retrieval model, each training sample is mapped into a vector with position information using the patch embedding layer at the input end of the large model DINOv2 to obtain the input vector of each training sample;

[0094] See also Figure 3 It is understood that this step may include:

[0095] ① Use the patch embedding layer (Patch layer) to evenly divide the image corresponding to each input training sample into K blocks; and map each block image into a D-dimensional vector;

[0096] As a specific example, for a small batch of input training sets, each training sample, i.e., sketch training sample and photo training sample, is an image. The Patch layer is a learnable linear projection layer, which will evenly divide the image corresponding to each input training sample into K blocks. For each of the K blocks, the block image will be mapped to a D-dimensional vector.

[0097] Wherein, K and D are natural numbers.

[0098] ②Add a learnable classification vector x class , which is used as the output of the retrieval model, and the input vector is constructed for each training sample as follows:

[0099]

[0100] Among them, the learnable classification vector x class When setting, it has initialization information, x class Contains multiple elements, corresponding to the confidence of each category. Through the model learning process, the output of the model will eventually be formed.

[0101] The image corresponding to the training sample is evenly divided into K blocks and mapped respectively. The vector after the j-th block image is mapped is D-dimensional; E pos Represents the relative position information of all elements in z0, in vector form.

[0102] 2) extracting features from the input vector of each training sample using N transformer blocks in turn, and outputting the output vector of each training sample through the multi-layer perceptron layer at the end of the large model DINOv2;

[0103] Among them, the weight branch and the low-rank matrix branch in the domain-independent module simultaneously receive the vector data of the first modality image training samples and the second modality image training samples; in the domain-dependent module, the weight branch is respectively connected in parallel with the two low-rank matrix branches and added, and its weight branch simultaneously receives the vector data of the first modality image training samples and the second modality image training samples, one low-rank matrix branch receives the vector data of the first modality image training samples, and the other low-rank matrix branch receives the vector data of the second modality image training samples; the weight matrix W in all weight branches is frozen; each low-rank matrix branch includes a low-rank matrix B and a low-rank matrix A connected in series; the dimension of the weight matrix W is d×k; the dimensions of the low-rank matrix B and the low-rank matrix A are d×r and r×k respectively, r is the rank, and the value of r meets the low rank requirement.

[0104] The multi-layer perceptron layer at the end of the large model DINOv2 is also called the MLP head layer, see Figure 3 The multi-layer perceptron head in .

[0105] As a specific example, since sketch retrieval belongs to cross-modal retrieval, the model must have a certain feature robustness, which requires the model to not only understand the similar semantics of sketches and photos (such as contours), but also distinguish their respective specific semantics (such as texture). Based on this, the present invention uses the DA-LoRA module and the DS-LoRA module to identify the shared features and specific features of the two types of modalities, respectively. The DA-LoRA module is designed to identify and utilize the common features of the two modalities, thereby promoting knowledge sharing, and the DS-LoRA module is designed to capture the specific features of a specific domain.

[0106] Specifically, the weight branch and the low-rank matrix branch in the DA-LoRA module simultaneously receive the vector data of the sketch training sample and the photo training sample; in the DS-LoRA module, the weight branch is respectively connected in parallel with the two low-rank matrix branches and added, and the weight branch simultaneously receives the vector data of the sketch training sample and the photo training sample, one low-rank matrix branch receives the vector data of the sketch training sample, and the other low-rank matrix branch receives the vector data of the photo training sample;

[0107] Specifically, for the first N-1 transformer blocks, the output z of any transformer block l is: z′ l =Attention(LN(z l-1 ))+DALoRA Attention (LN(z l-1 ))+z l-1 ,l=1,…,N-1;

[0108] z l =MLP(LN(z′ l )+DALoRA MLP (LN(z′ l ))+z′ l ),l=1,…,N-1;

[0109] Among them, DALoRA Attention Indicates the integration of DA-LoRA module in Attention module, DALoRA MLP It represents the integration of DA-LoRA module in MLP module, Attention represents the processing of Attention module, LN() represents the processing of linear layer, and MLP() represents the processing of MLP module.

[0110] The DA-LoRA module of the embodiment of the present invention is intended to learn the shared information between the two modalities of sketch and photo. Therefore, whether it is the Attention module and MLP module of the first N-1 transformer blocks, or the QKV (integrated DA-LoRA module) in the Attention module of the last Nth transformer block, they all receive the input of sketch and photo vectors at the same time.

[0111] For the last Nth transformer block, combine Figure 3 Understand that the output y of the model is:

[0112] z' N =Attention(LN(z N-1 ))+DSLoRA Attention (LN(z N-1 ))+z N-1 ;

[0113] z N =MLP(LN(z′ N )+DSLoRA MLP (LN(z′ N ))+z′ N );

[0114] Among them, DSLoRA Attention Indicates the integration of DS-LoRA module in Attention module, DSLoRA MLP Indicates the integration of DS-LoRA module in MLP module.

[0115] The output y of the model is the output vector. Specifically, y is Figure 3 The 768-dimensional vector output after the last MLP head layer in y (768 is related to the setting of the MLP head layer). class The results after changes in each layer of the model.

[0116] The DS-LoRA module of the embodiment of the present invention is designed to learn specific knowledge of a specific domain, and is mainly concentrated in the MLP module in the last transformer block. In addition to being input into the frozen weight matrix W, the sketch vector is input into the low-rank matrices A1 and B1 in one of the low-rank matrix branches, and the photo vector is input into the low-rank matrices A2 and B2 in another low-rank matrix branch. It can be seen that the DS-LoRA module distinguishes between the two modalities, and samples of different modalities enter different low-rank matrices, while the DA-LoRA module accepts both sketch and photo sample vectors without distinguishing between modalities.

[0117] The feature extraction process of the embodiment of the present invention should be understood in combination with the original feature extraction process of the large model DINOv2 and the description of the above-mentioned DA-LoRA module and DS-LoRA module.

[0118] S04, for each first modality image training sample, using its output vector, and the output vectors of the second modality image training samples of the same category and of different categories, calculate the value of its loss function, and obtain the value of the total loss function of all the first modality image training samples in the small batch;

[0119] Among them, for each first modality image training sample, the value of its loss function is calculated using its output vector, the output vectors of the second modality image training samples of the same category and different categories, and is expressed as follows:

[0120]

[0121] in, is the first modality image training sample The output vector, L i is the first modality image training sample The value of the loss function, For The output vector of the second modality image training sample of the same category, For Output vectors of the second modality image training samples of different categories, d(,) is the calculated Euclidean distance, and α is the interval hyperparameter;

[0122] As a specific example, Training samples for sketches The output vector, L i Training samples for sketches The value of the loss function, For The output vector of the training samples of photos of the same category, For Output vectors of photo training samples of different categories. The above loss function is a triplet loss, and the goal is that the Euclidean distance of the output vectors of sketches and photos of the same category is close enough, and the Euclidean distance of the output vectors of sketches and photos of different categories is far enough. α is an indicator to measure "far and near", which can be set as needed. The optional value of the embodiment of the present invention is 0.2.

[0123] Among them, the value of the total loss function of all first modality image training samples in the small batch is obtained, which is expressed as:

[0124]

[0125] Wherein, M is the number of all first modality image training samples in the small batch. In a specific example, M is the number of all sketch training samples in the small batch.

[0126] Through the above processing, the value of the total loss function of the current round of training can be obtained.

[0127] S05, based on the gradient of the total loss function, optimize the low-rank matrices in each domain-independent module and domain-dependent module to complete the current round of training;

[0128] As a specific example, this step is to optimize the low-rank matrices A and B in each DA-LoRA module and DS-LoRA module based on the gradient of the total loss function to complete the current round of training.

[0129] The gradient of the total loss function is obtained based on the value of the total loss function of the current round of training and the value of the total loss function of the previous round of training.

[0130] As a specific example, based on the gradient of the total loss function, the low-rank matrices A and B in each DA-LoRA module and DS-LoRA module are optimized using the Adam optimizer. The specific optimization process can be understood by referring to the existing model training process and will not be described in detail here.

[0131] Among them, the learning rate during training can be 1e-6.

[0132] S06, repeat multiple rounds of training until a preset maximum number of training rounds is reached to obtain a pre-trained retrieval model.

[0133] S03 to S05 are to perform a round of training. If the preset maximum number of training rounds max_epochs is not reached, S03 to S05 are continued to be performed until the preset maximum number of training rounds max_epochs is reached, and the training is stopped to obtain a trained retrieval model.

[0134] The preset maximum number of training rounds max_epochs can be set as needed, for example, to 20.

[0135] The embodiment of the present invention uses low-rank technology instead of directly training a large model to explore the field of cross-modal image retrieval such as zero-sample sketch retrieval. The overall model is based on the Vision Transformer architecture of the large model DINOv2. Domain-independent modules and domain-related modules are designed for the similarities and unique points of two different modal images, and the corresponding layers in some modules of DINOv2 are replaced. Specifically, a low-rank matrix constructed by low-rank technology is added to the original weight matrix as a low-rank branch, so that the low-rank technology can better adapt to the zero-sample image retrieval task belonging to cross-modal retrieval, and increase the robustness of the model. The original parameters of DINOv2 are frozen during the training process, and only the low-rank matrices in the domain-independent module and the domain-related module are optimized. The present invention introduces large-model low-rank adaptation technology into the zero-sample image retrieval task, which can reduce the model training parameters while maintaining or even better than the accuracy of training a complete large model.

[0136] In a specific example taking sketch retrieval as an example, the present invention is to give a sketch retrieval task dataset where y represents the i-th sample, i Denotes the corresponding category label value. E and P represent the sketch and photo modalities respectively. The goal of zero-shot sketch retrieval is to s (i.e., training set) to train a model with excellent discriminative representation, and then t The zero-sample setting requires D s and D t The dataset contains completely different categories. In the inference phase, the sketch is used as a query to retrieve photos belonging to the same category in the photo library (which can be the photo set in the test set). Therefore, the goal of zero-shot sketch retrieval is to learn a cross-modal representation F with good generalization. θ (·) to avoid the overfitting problem and achieve a high retrieval accuracy.

[0137] The original large model DINOv2 will directly train the weight matrix W. Taking the linear module as an example, the weight matrix W is 3072*768 in size, r(W), i.e., the rank of the weight matrix, is very large, and the complete training consumes more resources. The present invention freezes the weight matrix W and trains two low-rank matrices A and B instead. The rank of A is 3072*64, and the rank of B is 64*768. The rank A and rank B are both relatively small. At the same time, the retrieval effect achieved is better than that of directly training the weight matrix W.

[0138] The domain-independent DA-LoRA module receives data from both the sketch and photo domains to identify and utilize the common features of the two modalities. The domain-dependent DS-LoRA module has two low-rank matrix branches that each receive data from one modality, thereby capturing the specific features of a particular domain to ensure that LoRA technology can better fit downstream sketch retrieval tasks and improve retrieval accuracy.

[0139] It can be seen that for cross-modal image retrieval, the present invention can reduce model training parameters while ensuring the accuracy of model training.

[0140] After obtaining the pre-trained retrieval model, the present invention uses a test set to verify the performance of the retrieval model, and if the verification passes, it can be used for formal retrieval.

[0141] The test set verification process includes:

[0142] 1) Inputting the test set into the pre-trained retrieval model to obtain an output vector of each test sample;

[0143] The test set contains first modality image test samples and second modality image test samples, and the categories are known;

[0144] 2) For any first modality image test sample, the output vector thereof is compared with the output vectors of the second modality image test samples of each category, and a preset number of second modality image test samples that meet the low difference requirement are determined to form a neighboring set of the first modality image test sample; the number of second modality image test samples in the neighboring set that are consistent with the category of the first modality image test sample is determined, so as to obtain the accuracy of the retrieval model corresponding to the first modality image test sample;

[0145] In this step, the output vectors of the second modality image test samples of each category can be saved for use in the formal retrieval stage.

[0146] Wherein, for any first modality image test sample, the output vector thereof is compared with the output vectors of the second modality image test samples of each category, and a preset number of second modality image test samples that meet the low difference requirement are determined to form a neighboring set of the first modality image test sample, including:

[0147] For any first modality image test sample, the Euclidean distance is calculated for its output vector and the output vectors of the second modality image test samples of each category, and the samples are sorted from small to large according to the Euclidean distance, and a preset number of second modality image test samples ranked first are taken to form a neighboring set.

[0148] Similarly, step S3 of the present invention also calculates the Euclidean distance between the output vector of the first modality image to be retrieved and the output vector of the second modality image test samples of each category, sorts them from small to large according to the Euclidean distance, and takes a preset number of second modality image test samples in the front sorting to form a retrieval set, thereby obtaining a retrieval result. The second modality image test samples in the retrieval set are considered to be consistent with or close to the category of the first modality image to be retrieved, that is, they meet the low difference requirement.

[0149] The preset number can be selected as needed, for example, 100 or 200.

[0150] Among them, after determining the number of second modality image test samples in the neighboring set that are consistent with the category of the first modality image test sample, the retrieval model accuracy corresponding to the first modality image test sample can be obtained by using the ratio of this number to a preset number.

[0151] 3) The retrieval model accuracy corresponding to all first modality image test samples is used to obtain the final accuracy of the retrieval model, and verification is passed when the final accuracy meets the requirements.

[0152] Specifically, the retrieval model accuracies corresponding to all first modality image test samples may be averaged to obtain the final accuracy of the retrieval model.

[0153] The quantitative indicators of the final accuracy can be mAP and Prec.

[0154] It is understandable that if the final accuracy does not meet the requirements, the verification fails, and retraining and verification can be performed again until the verification passes.

[0155] The embodiment of the present invention verifies the trained retrieval model again using a test set. When the accuracy requirement is met, a retrieval model with application performance can be obtained. When the accuracy requirement is not met, it can be trained and adjusted again so that the retrieval model meets the requirement, thereby improving the accuracy of the retrieval stage.

[0156] In a specific example, the embodiment of the present invention uses the trained retrieval model to obtain multiple photos of the same or similar categories of the sketch to be retrieved, thereby completing the sketch retrieval. This method improves the inference speed while maintaining or even exceeding the accuracy of conventional methods.

[0157] Based on the above method embodiment, the present invention also provides a zero-sample cross-modal image retrieval device based on low-rank adaptation, see Figure 4 , the device comprises:

[0158] A retrieval model acquisition module is used to acquire a retrieval model; wherein, the retrieval model is obtained by replacing the corresponding layers in the transformer block of the large model DINOv2 with pre-designed domain-independent modules and domain-dependent modules by using a preset low-rank technology, thereby adding parallel low-rank matrix branches to the original weights, and freezing the original network parameter gradients, and then training with a training set containing first modality image training samples and second modality image training samples, and verifying with a test set after the training is completed; the first modality and the second modality are different modalities, and the categories of samples in the test set and the training set are different;

[0159] A retrieval model processing module, used for inputting the first modality image to be retrieved into the retrieval model to obtain a corresponding output vector;

[0160] The retrieval set determination module is used to compare the output vectors of the first modality image to be retrieved with the output vectors of the second modality image test samples of each category determined during the test set verification process, and determine a preset number of second modality image test samples that meet the low difference requirement to form a retrieval set.

[0161] In one optional implementation, the training process of the retrieval model includes:

[0162] Get the large model DINOv2 whose network parameters have been pre-trained;

[0163] Using the preset low-rank technology, the corresponding layers in some preset modules in the first N-1 transformer blocks of the large model DINOv2 are replaced with the domain-independent modules, the corresponding layers in some preset modules in the Nth transformer block are replaced with the domain-independent modules and the domain-related modules, and the corresponding layers in some preset modules in the Nth transformer block are replaced with the domain-related modules, so as to obtain a constructed retrieval model; wherein the domain-independent modules and the domain-related modules are obtained by retaining the original weights in the replaced layers and adding low-rank matrix branches for parallel addition, and the low-rank matrix branches are composed of low-rank matrices constructed by the preset low-rank technology connected in series;

[0164] Freeze the original network parameter gradient of the large model DINOv2, and for the current round of training, input a small batch of training sets containing first modality image training samples and second modality image training samples into the built retrieval model, and obtain the output vector of each training sample through feature extraction processing of the retrieval model; wherein the categories of the training samples in the training set are known;

[0165] For each first modality image training sample, use its output vector, the output vectors of the second modality image training samples of the same category and different categories to calculate the value of its loss function, and obtain the value of the total loss function of all the first modality image training samples in the small batch;

[0166] Based on the gradient of the total loss function, the low-rank matrices in each domain-independent module and domain-dependent module are optimized to complete the current round of training;

[0167] Repeat multiple rounds of training until the preset maximum number of training rounds is reached to obtain a pre-trained retrieval model.

[0168] In an optional implementation, when the preset low-rank technology is a low-rank adaptive LoRA technology, the preset low-rank technology is used to replace the corresponding layers in some preset modules in the first N-1 transformer blocks of the large model DINOv2 with the domain-independent modules, and the corresponding layers in some preset modules in the Nth transformer block are replaced with the domain-independent modules and the domain-related modules, and the corresponding layers in some preset modules in the Nth transformer block are replaced with the domain-related modules, to obtain a completed retrieval model, including:

[0169] For the N transformer blocks of the large model DINOv2, the low-rank adaptive LoRA technology is used to replace the query-key-value layer, the projection layer in the multi-head attention module in the first N-1 transformer blocks, and the two linear layers in the linear module with the domain-independent module, the query-key-value layer in the multi-head attention module in the Nth transformer block is replaced with the domain-independent module, the projection layer is replaced with the domain-related module, and the two linear layers in the linear module in the Nth transformer block are replaced with the domain-related module, so as to obtain a completed retrieval model.

[0170] In an optional implementation, for the current round of training, a small batch of training sets containing first modality image training samples and second modality image training samples are input into the constructed retrieval model, and an output vector of each training sample is obtained through feature extraction processing of the retrieval model, including:

[0171] After the small batch of the current round of training is input into the retrieval model, each training sample is mapped into a vector with position information using the patch embedding layer at the input end of the large model DINOv2 to obtain the input vector of each training sample;

[0172] The input vector of each training sample is feature extracted in turn using N transformer blocks, and the output vector of each training sample is output through the multi-layer perceptron layer at the end of the large model DINOv2; wherein, the weight branch and the low-rank matrix branch in the domain-independent module simultaneously receive the vector data of the first modality image training sample and the second modality image training sample; in the domain-dependent module, the weight branch is respectively connected in parallel with the two low-rank matrix branches and added, and its weight branch simultaneously receives the vector data of the first modality image training sample and the second modality image training sample, one low-rank matrix branch receives the vector data of the first modality image training sample, and the other low-rank matrix branch receives the vector data of the second modality image training sample; the weight matrix W in all weight branches is frozen; each low-rank matrix branch includes a low-rank matrix B and a low-rank matrix A connected in series; the dimension of the weight matrix W is d×k; the dimensions of the low-rank matrix B and the low-rank matrix A are d×r and r×k respectively, r is the rank, and the value of r meets the low-rank requirement.

[0173] In an optional implementation, each training sample is mapped into a vector with position information using the patch embedding layer at the input end of the large model DINOv2 to obtain an input vector of each training sample, including:

[0174] The patch embedding layer is used to evenly divide the image corresponding to each input training sample into K blocks; and each block image is mapped into a D-dimensional vector;

[0175] Add a learnable classification vector x class , which is used as the output of the retrieval model, and the input vector is constructed for each training sample as follows:

[0176]

[0177] in, The image corresponding to the training sample is evenly divided into K blocks and mapped respectively. The vector after the j-th block image is mapped is D-dimensional; E pos Represents the relative position information of all elements in z0, in vector form.

[0178] In an optional implementation manner, for each first modality image training sample, the output vector thereof, and the output vectors of the second modality image training samples of the same category and of different categories are used to calculate the value of the loss function thereof, which is expressed as follows:

[0179]

[0180] in, is the first modality image training sample The output vector, L i is the first modality image training sample The value of the loss function, For The output vector of the second modality image training sample of the same category, For Output vectors of the second modality image training samples of different categories, d(,) is the calculated Euclidean distance, and α is the interval hyperparameter;

[0181] The value of the total loss function of all first modality image training samples of the small batch is expressed as follows:

[0182]

[0183] Where M is the number of all first modality image training samples in this small batch.

[0184] In an optional implementation, the low-rank matrices in each domain-independent module and domain-dependent module are optimized based on the gradient of the total loss function to complete the current round of training, which is implemented using the Adam optimizer.

[0185] In an optional implementation manner, the test set verification process includes:

[0186] Inputting the test set into the pre-trained retrieval model to obtain an output vector of each test sample; wherein the test set contains a first modality image test sample and a second modality image test sample, and the category is known;

[0187] For any first modality image test sample, the output vector thereof is compared with the output vectors of the second modality image test samples of each category, and a preset number of second modality image test samples that meet the low difference requirement are determined to form a neighboring set of the first modality image test sample; the number of second modality image test samples in the neighboring set that are consistent with the category of the first modality image test sample is determined, so as to obtain the accuracy of the retrieval model corresponding to the first modality image test sample;

[0188] The retrieval model accuracy corresponding to all first modality image test samples is used to obtain the final accuracy of the retrieval model, and verification is passed when the final accuracy meets the requirements.

[0189] In an optional implementation manner, for any first modality image test sample, the output vector thereof is compared with the output vectors of the second modality image test samples of each category, and a preset number of second modality image test samples that meet the low difference requirement are determined to form a neighboring set of the first modality image test sample, including:

[0190] For any first modality image test sample, the Euclidean distance is calculated for its output vector and the output vectors of the second modality image test samples of each category, and the samples are sorted from small to large according to the Euclidean distance, and a preset number of second modality image test samples ranked first are taken to form a neighboring set.

[0191] For the specific content of the device, please refer to the relevant content of the zero-sample cross-modal image retrieval method based on low-rank adaptation provided above, which will not be described in detail here.

[0192] The present invention introduces large-model low-rank adaptation technology into zero-shot cross-modal image retrieval tasks such as zero-shot sketch retrieval, which can reduce the model training parameters while maintaining or even exceeding the accuracy of training a complete large model.

[0193] In order to verify the effect of the solution of the embodiment of the present invention, relevant simulation experiment data are given below for the above-mentioned specific examples.

[0194] Firstly, the effect of the self-supervised visual large model DINOv2 is evaluated on the public zero-shot sketch retrieval dataset Sketchy Ext based on triplet loss.

[0195] The Sketchy Ext dataset contains 125 categories, 75,471 sketches and 12,500 photos. Liu et al. 2019 collected an additional 60,502 natural photos, for a total of 73,002 photos.

[0196] In order to verify the effectiveness of the present invention, the evaluation indicators of other retrieval methods are used here, including: mean average accuracy (mAP) and precision (Prec). mAP and Prec are indicators for comprehensively evaluating the overall performance of the retrieval system. The larger the value, the higher the retrieval effect. The experimental results are shown in Table 1.

[0197] Table 1 Experimental results of zero-shot sketch retrieval based on domain adaptation

[0198]

[0199] Here, the LoRA mode only means that only the DA-LoRA module of the present invention is used without a DS-LoRA module.

[0200] As can be seen from Table 1, simple LoRA (LoRA mode only) can achieve a level similar to that of retraining DINOv2 with very few parameters, but the overall performance is still not as good as fully training the entire model. It is worth noting that the latest zero-shot sketch retrieval methods perform lower than or similar to the DINOv2 model. For example, in 2013, Aneeshan Sain et al. proposed the retrieval method CLIP-AT, whose mAP@200 and Prec@200 on this dataset were 0.723 and 0.725 respectively; in 2014, Subhadeep Koley et al. proposed TTI, with mAP@200 and Prec@200 of 0.746 and 0.747 respectively. These methods are all to fully train a model with a large number of parameters. On the other hand, compared with the fully trained DINOv2 model, the method of the present invention uses about 12% of the parameters to achieve the best results on the Sketchy Ext dataset. This further demonstrates the superiority of the method of the present invention. While reducing the training parameters and time overhead, the method of the present invention exceeds the strategies of simple LoRA and full parameter training.

[0201] In summary, the accuracy of the method of the present invention exceeds that of the complete training of large models and the simple application of LoRA technology. At the same time, the method of the present invention is not targeted at a specific model architecture, which makes it easy to deploy in real-world scenarios.

[0202] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0203] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A zero-shot cross-modal image retrieval method based on low-rank adaptation, characterized in that: include: Obtain a retrieval model; wherein, the retrieval model is obtained by replacing the corresponding layers in the transformer block of the large model DINOv2 with pre-designed domain-independent modules and domain-dependent modules by using a preset low-rank technology, thereby adding parallel low-rank matrix branches to the original weights, and freezing the original network parameter gradients, and then training with a training set containing first modality image training samples and second modality image training samples, and verifying with a test set after the training is completed; the first modality and the second modality are different modalities, and the categories of samples in the test set and the training set are different; Inputting the first modality image to be retrieved into the retrieval model to obtain a corresponding output vector; The output vector of the first modality image to be retrieved and the output vector of each category of the second modality image test samples determined in the test set verification process are compared respectively, and a preset number of second modality image test samples that meet the low difference requirement are determined to form a retrieval set.

2. The zero-shot cross-modal image retrieval method based on low-rank adaptation according to claim 1, characterized in that: The training process of the retrieval model includes: Get the large model DINOv2 whose network parameters have been pre-trained; Using the preset low-rank technology, the corresponding layers in some preset modules in the first N-1 transformer blocks of the large model DINOv2 are replaced with the domain-independent modules, the corresponding layers in some preset modules in the Nth transformer block are replaced with the domain-independent modules and the domain-related modules, and the corresponding layers in some preset modules in the Nth transformer block are replaced with the domain-related modules, so as to obtain a constructed retrieval model; wherein the domain-independent modules and the domain-related modules are obtained by retaining the original weights in the replaced layers and adding low-rank matrix branches for parallel addition, and the low-rank matrix branches are composed of low-rank matrices constructed by the preset low-rank technology connected in series; Freeze the original network parameter gradient of the large model DINOv2, and for the current round of training, input a small batch of training sets containing first modality image training samples and second modality image training samples into the built retrieval model, and obtain the output vector of each training sample through feature extraction processing of the retrieval model; wherein the categories of the training samples in the training set are known; For each first modality image training sample, use its output vector, the output vectors of the second modality image training samples of the same category and different categories to calculate the value of its loss function, and obtain the value of the total loss function of all the first modality image training samples in the small batch; Based on the gradient of the total loss function, the low-rank matrices in each domain-independent module and domain-dependent module are optimized to complete the current round of training; Repeat multiple rounds of training until the preset maximum number of training rounds is reached to obtain a pre-trained retrieval model.

3. The zero-shot cross-modal image retrieval method based on low-rank adaptation according to claim 2, characterized in that: When the preset low-rank technology is the low-rank adaptive LoRA technology, the preset low-rank technology is used to replace the corresponding layers in some preset modules in the first N-1 transformer blocks of the large model DINOv2 with the domain-independent modules, replace the corresponding layers in some preset modules in the Nth transformer block with the domain-independent modules and the domain-related modules, and replace the corresponding layers in some preset modules in the Nth transformer block with the domain-related modules, to obtain a constructed retrieval model, including: For the N transformer blocks of the large model DINOv2, the low-rank adaptive LoRA technology is used to replace the query-key-value layer, the projection layer in the multi-head attention module in the first N-1 transformer blocks, and the two linear layers in the linear module with the domain-independent module, the query-key-value layer in the multi-head attention module in the Nth transformer block is replaced with the domain-independent module, the projection layer is replaced with the domain-related module, and the two linear layers in the linear module in the Nth transformer block are replaced with the domain-related module, so as to obtain a completed retrieval model.

4. The zero-shot cross-modal image retrieval method based on low-rank adaptation according to claim 3, characterized in that: For the current round of training, a small batch of training sets containing first modality image training samples and second modality image training samples is input into the constructed retrieval model, and an output vector of each training sample is obtained through feature extraction processing of the retrieval model, including: After the small batch of the current round of training is input into the retrieval model, each training sample is mapped into a vector with position information using the patch embedding layer at the input end of the large model DINOv2 to obtain the input vector of each training sample; The input vector of each training sample is feature extracted in turn using N transformer blocks, and the output vector of each training sample is output through the multi-layer perceptron layer at the end of the large model DINOv2; wherein, the weight branch and the low-rank matrix branch in the domain-independent module simultaneously receive the vector data of the first modality image training sample and the second modality image training sample; in the domain-dependent module, the weight branch is respectively connected in parallel with the two low-rank matrix branches and added, and its weight branch simultaneously receives the vector data of the first modality image training sample and the second modality image training sample, one low-rank matrix branch receives the vector data of the first modality image training sample, and the other low-rank matrix branch receives the vector data of the second modality image training sample; the weight matrix W in all weight branches is frozen; each low-rank matrix branch includes a low-rank matrix B and a low-rank matrix A connected in series; the dimension of the weight matrix W is d×k; the dimensions of the low-rank matrix B and the low-rank matrix A are d×r and r×k respectively, r is the rank, and the value of r meets the low-rank requirement.

5. The zero-shot cross-modal image retrieval method based on low-rank adaptation according to claim 4, characterized in that: Each training sample is mapped into a vector with position information using the patch embedding layer at the input end of the large model DINOv2 to obtain the input vector of each training sample, including: The patch embedding layer is used to evenly divide the image corresponding to each input training sample into K blocks; and each block image is mapped into a D-dimensional vector; Add a learnable classification vector x class , which is used as the output of the retrieval model, and the input vector is constructed for each training sample as follows: in, The image corresponding to the training sample is evenly divided into K blocks and mapped respectively. The vector after the j-th block image is mapped is D-dimensional; E pos Represents the relative position information of all elements in z0, in vector form.

6. The zero-shot cross-modal image retrieval method based on low-rank adaptation according to claim 4, characterized in that: For each first modality image training sample, the output vector and the output vectors of the second modality image training samples of the same category and different categories are used to calculate the value of the loss function, which is expressed as: in, is the first modality image training sample The output vector, L i is the first modality image training sample The value of the loss function, For The output vector of the second modality image training sample of the same category, For Output vectors of the second modality image training samples of different categories, d(,) is the calculated Euclidean distance, and α is the interval hyperparameter; The value of the total loss function of all first modality image training samples of the small batch is expressed as follows: Where M is the number of all first modality image training samples in this small batch.

7. The zero-shot cross-modal image retrieval method based on low-rank adaptation according to claim 4, characterized in that: The low-rank matrices in each domain-independent module and domain-dependent module are optimized based on the gradient of the total loss function to complete the current round of training, which is implemented using the Adam optimizer.

8. The zero-shot cross-modal image retrieval method based on low-rank adaptation according to claim 2, characterized in that: The test set verification process includes: Inputting the test set into the pre-trained retrieval model to obtain an output vector of each test sample; wherein the test set contains a first modality image test sample and a second modality image test sample, and the category is known; For any first modality image test sample, the output vector thereof is compared with the output vectors of the second modality image test samples of each category, and a preset number of second modality image test samples that meet the low difference requirement are determined to form a neighboring set of the first modality image test sample; the number of second modality image test samples in the neighboring set that are consistent with the category of the first modality image test sample is determined, so as to obtain the accuracy of the retrieval model corresponding to the first modality image test sample; The retrieval model accuracy corresponding to all first modality image test samples is used to obtain the final accuracy of the retrieval model, and verification is passed when the final accuracy meets the requirements.

9. The zero-shot cross-modal image retrieval method based on low-rank adaptation according to claim 8, characterized in that: For any first modality image test sample, the output vector thereof is compared with the output vectors of the second modality image test samples of each category, and a preset number of second modality image test samples that meet the low difference requirement are determined to form a neighboring set of the first modality image test sample, including: For any first modality image test sample, the Euclidean distance is calculated for its output vector and the output vectors of the second modality image test samples of each category, and the samples are sorted from small to large according to the Euclidean distance, and a preset number of second modality image test samples ranked first are taken to form a neighboring set.

10. A zero-sample cross-modal image retrieval device based on low-rank adaptation, characterized in that: include: A retrieval model acquisition module is used to acquire a retrieval model; wherein, the retrieval model is obtained by replacing the corresponding layers in the transformer block of the large model DINOv2 with pre-designed domain-independent modules and domain-dependent modules by using a preset low-rank technology, thereby adding parallel low-rank matrix branches to the original weights, and freezing the original network parameter gradients, and then training with a training set containing first modality image training samples and second modality image training samples, and verifying with a test set after the training is completed; the first modality and the second modality are different modalities, and the categories of samples in the test set and the training set are different; A retrieval model processing module, used for inputting the first modality image to be retrieved into the retrieval model to obtain a corresponding output vector; The retrieval set determination module is used to compare the output vectors of the first modality image to be retrieved with the output vectors of the second modality image test samples of each category determined during the test set verification process, and determine a preset number of second modality image test samples that meet the low difference requirement to form a retrieval set.

Images