A zero-shot cross-modal image retrieval method and device based on low-rank adaptation

Abstract

The application discloses a zero sample cross-modal image retrieval method and device based on low rank adaptation, and the method comprises the following steps: inputting a first modal image to be retrieved into a retrieval model to obtain an output vector; comparing the output vector with output vectors of predetermined second modal image test samples of various categories, and screening out second modal image test samples with low differences to form a retrieval set. The retrieval model is obtained by using a preset low rank technology, replacing the corresponding layers in the transformer block of a large model DINOv2 with domain-independent and domain-related modules to increase the parallel low rank matrix branch, and freezing the original network parameter gradient training and verification. The application is aimed at the similar points and unique points of two different modalities, and the domain-independent and domain-related modules replace part of the layers in DINOv2, so that the low rank technology can better adapt to the zero sample cross-modal image retrieval task, and the model robustness is increased. During training, the original parameters are frozen, and only the low rank matrix is optimized, so that the training parameters are reduced, and better accuracy can be provided.

Description

Technical Field

[0001] This invention belongs to the field of image retrieval, specifically relating to a zero-sample cross-modal image retrieval method and apparatus based on low-rank adaptation. Background Technology

[0002] Existing zero-shot image retrieval methods are mainly divided into two types: knowledge distillation-based methods and large model-based methods.

[0003] 1) Knowledge distillation-based methods (Liu et al. 2019; Tian et al. 2022) aim to enable student models to retain the knowledge learned by the teacher model during the pre-training phase, preventing it from being forgotten in downstream tasks. These methods typically employ a training approach that fixes the gradient of the teacher model and directionally transmits knowledge to the student model, which to some extent limits the model's generalization ability. Liu et al. (2023) proposed a method for bidirectional knowledge alignment between teacher and student models, where the teacher can dynamically fine-tune the knowledge transmitted to the student in the next stage based on the cross-modal knowledge learned by the student model, to better adapt to the context of multimodal retrieval. However, the above methods require training two or more models simultaneously, and only one model is used during inference, resulting in poor training efficiency.

[0004] 2) Large-Model-Based Approaches (Sain et al. 2023; Koley et al. 2024): In recent years, various large-scale language-visual models have emerged. These aim to leverage the superior performance of contrastive language-image pre-trained large models (CLIPs) in multimodal learning to solve image retrieval problems. CLIP models are pre-trained on ultra-large-scale image-text datasets. Benefiting from a large number of parameters and training data, they possess rich feature representations in downstream tasks, demonstrating strong generalization capabilities. This allows the model to fully understand the local similarity semantics between the image to be retrieved and another modal image being retrieved in image retrieval tasks. Although large models perform well in various fields, existing methods for zero-shot image retrieval often directly use large models to utilize their rich feature representations, which requires a large amount of training data and hardware resources. Furthermore, the large number of parameters leads to slow inference speeds. Limited by the enormous number of parameters, they are difficult to directly use for training in downstream tasks. Therefore, in the field of image retrieval, how to efficiently fine-tune these large models is becoming increasingly important. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a zero-sample cross-modal image retrieval method and apparatus based on low-rank adaptation. The technical problem to be solved by this invention is achieved through the following technical solution:

[0006] In a first aspect, embodiments of the present invention provide a zero-sample cross-modal image retrieval method based on low-rank adaptation, comprising:

[0007] A retrieval model is obtained; wherein the retrieval model is obtained by using a preset low-rank technique to replace the corresponding layers in the transformer block of the large model DINOv2 with pre-designed domain-independent modules and domain-dependent modules, thereby adding parallel low-rank matrix branches to the original weights, freezing the gradients of the original network parameters, training with a training set containing training samples of the first modality and training samples of the second modality, and verifying the results with a test set after training; the first modality and the second modality are different modalities, and the categories of the samples in the test set and the training set are different;

[0008] The first modality image to be retrieved is input into the retrieval model to obtain the corresponding output vector;

[0009] The output vector of the first modality image to be retrieved and the output vectors of the second modality image test samples of each category determined during the test set verification process are compared to determine the preset number of second modality image test samples that meet the low difference requirement, thus forming the retrieval set.

[0010] In one embodiment of the present invention, the training process of the retrieval model includes:

[0011] Obtain the pre-trained large model DINOv2 with network parameters;

[0012] Using a pre-defined low-rank technique, the corresponding layers in the pre-defined modules within the first N-1 transformer blocks of the large model DINOv2 are replaced with the domain-independent modules. The corresponding layers in the pre-defined modules within the Nth transformer block are replaced with both the domain-independent and domain-related modules. The corresponding layers in the pre-defined modules within the Nth transformer block are replaced with the domain-related modules, resulting in a completed retrieval model. The domain-independent and domain-related modules are obtained by retaining the original weights within the replaced layers and adding low-rank matrix branches for parallel summation. These low-rank matrix branches are constructed by concatenating low-rank matrices built using the pre-defined low-rank technique.

[0013] The gradients of the original network parameters of the large model DINOv2 are frozen. For the current training round, a small batch of the training set containing the first modality image training samples and the second modality image training samples is input into the completed retrieval model. Through feature extraction processing of the retrieval model, the output vector of each training sample is obtained; wherein, the category of the training samples in the training set is known.

[0014] For each first modality image training sample, its output vector and the output vectors of second modality image training samples of the same and different classes are used to calculate the value of its loss function, and the value of the total loss function of all first modality image training samples in this mini-batch is obtained.

[0015] Based on the gradient of the total loss function, optimize the low-rank matrices in each domain-independent and domain-dependent module to complete the current round of training;

[0016] Repeat the training multiple times until the preset maximum number of training rounds is reached to obtain the pre-trained retrieval model.

[0017] In one embodiment of the present invention, when the preset low-rank technique is a low-rank adapted LoRA technique, the step of using the preset low-rank technique to replace the corresponding layers in the preset modules of the first N-1 transformer blocks of the large model DINOv2 with the domain-independent modules, replacing the corresponding layers in the preset modules of the Nth transformer block with the domain-independent modules and the domain-dependent modules, and replacing the corresponding layers in the preset modules of the Nth transformer block with the domain-dependent modules, results in a completed retrieval model, including:

[0018] For the N transformer blocks of the large model DINOv2, the low-rank adaptive LoRA technique is used to replace the query-key-value layer and projection layer in the multi-head attention module and the two linear layers in the linear module in the first N-1 transformer blocks with the domain-independent module. The query-key-value layer in the multi-head attention module in the Nth transformer block is replaced with the domain-independent module, the projection layer is replaced with the domain-related module, and the two linear layers in the linear module in the Nth transformer block are replaced with the domain-related module, thus obtaining the completed retrieval model.

[0019] In one embodiment of the present invention, for the current round of training, a small batch of the training set containing training samples of the first modality and training samples of the second modality is input into the constructed retrieval model. Through feature extraction processing of the retrieval model, the output vector of each training sample is obtained, including:

[0020] After inputting the small batch of training in the current round into the retrieval model, the patch embedding layer at the input end of the large model DINOv2 is used to map each training sample into a vector with position information, thus obtaining the input vector of each training sample.

[0021] The input vectors of each training sample are sequentially processed using N transformer blocks for feature extraction, and the output vectors of each training sample are output through the multilayer perceptron layer at the end of the large model DINOv2. The weight branch and low-rank matrix branch in the domain-independent module simultaneously receive vector data from the first and second modal image training samples. In the domain-related module, the weight branch is connected in parallel with two low-rank matrix branches and summed. The weight branch simultaneously receives vector data from the first and second modal image training samples, while one low-rank matrix branch receives vector data from the first modal image training sample, and the other receives vector data from the second modal image training sample. The weight matrix W in all weight branches is frozen. Each low-rank matrix branch includes a concatenated low-rank matrix B and a low-rank matrix A. The dimension of the weight matrix W is d×k. The dimensions of the low-rank matrix B and low-rank matrix A are d×r and r×k, respectively, where r is the rank, and the value of r satisfies the low-rank requirement.

[0022] In one embodiment of the present invention, each training sample is mapped to a vector with positional information using a patch embedding layer at the input of the large model DINOv2, resulting in the input vector of each training sample, including:

[0023] The patch embedding layer is used to uniformly divide the image corresponding to each input training sample into K blocks; and each block image is mapped to a D-dimensional vector.

[0024] Add a learnable classification vector x class This is used as the output of the retrieval model, and an input vector is constructed for each training sample as follows:

[0025]

[0026] in, After the images corresponding to the training samples are uniformly divided into K blocks and mapped respectively, the vector of the mapped j-th block image is D-dimensional; E pos This represents the relative position information of all elements in z0, in vector form.

[0027] In one embodiment of the present invention, for each first modality image training sample, the loss function is calculated using its output vector and the output vectors of second modality image training samples of the same and different classes, expressed by the formula:

[0028]

[0029] in, Training samples for the first modality image The output vector, L i Training samples for the first modality image The value of the loss function, To and The output vector of the training samples of the second modality of the same category, To and The output vectors of training samples of second modality images of different categories, d(,) is used to calculate the Euclidean distance, and α is the margin hyperparameter;

[0030] The value of the total loss function obtained for all first-modality image training samples in this mini-batch is expressed by the formula:

[0031]

[0032] Where M is the number of all first-modality image training samples in this mini-batch.

[0033] In one embodiment of the present invention, the low-rank matrix in each domain-independent module and domain-dependent module is optimized based on the gradient of the total loss function to complete the current round of training, which is achieved using the Adam optimizer.

[0034] In one embodiment of the present invention, the test set verification process includes:

[0035] The test set is input into the pre-trained retrieval model to obtain the output vector of each test sample; wherein the test set contains first modality image test samples and second modality image test samples, and the categories are known;

[0036] For any first modality image test sample, its output vector is compared with the output vectors of second modality image test samples of each category to determine a preset number of second modality image test samples that meet the low difference requirement, which constitute the neighborhood set of the first modality image test sample; the number of second modality image test samples in the neighborhood set that are consistent with the category of the first modality image test sample is determined, thereby obtaining the retrieval model accuracy corresponding to the first modality image test sample;

[0037] The final accuracy of the retrieval model is obtained by using the accuracy of the retrieval model corresponding to all the first modality image test samples. The model is verified when the final accuracy meets the requirements.

[0038] In one embodiment of the present invention, the step of comparing the difference between the output vector of any first modality image test sample and the output vectors of second modality image test samples of each category to determine a preset number of second modality image test samples that meet the low difference requirement, thereby forming a neighborhood set of the first modality image test sample, includes:

[0039] For any first modality image test sample, calculate the Euclidean distance between its output vector and the output vectors of the second modality image test samples of each category. Sort them in ascending order of Euclidean distance, and take a preset number of second modality image test samples at the top of the sort to form a neighborhood set.

[0040] Secondly, embodiments of the present invention provide a zero-sample cross-modal image retrieval device based on low-rank adaptation, comprising:

[0041] A retrieval model acquisition module is used to acquire a retrieval model. The retrieval model is obtained by using a preset low-rank technique to replace the corresponding layers in the transformer block of the large model DINOv2 with pre-designed domain-independent and domain-dependent modules, thereby adding parallel low-rank matrix branches to the original weights. After freezing the gradients of the original network parameters, the model is trained using a training set containing training samples of the first and second modalities, and validated using a test set after training. The first and second modalities are different modalities, and the categories of samples in the test set and the training set are different.

[0042] The retrieval model processing module is used to input the first modality image to be retrieved into the retrieval model and obtain the corresponding output vector;

[0043] The retrieval set determination module is used to compare the output vector of the first modality image to be retrieved with the output vector of the second modality image test samples of each category determined during the test set verification process, and determine a preset number of second modality image test samples that meet the low difference requirement to form the retrieval set.

[0044] The beneficial effects of this invention are:

[0045] This invention explores cross-modal image retrieval tasks such as zero-shot sketch retrieval using low-rank techniques instead of directly training a large model. The overall model is based on the Vision Transformer architecture of the large model DINOv2. Domain-independent and domain-related modules are designed to address the similarities and uniqueness of images from two different modalities. Corresponding layers in some DINOv2 modules are replaced, specifically by adding low-rank matrices constructed using low-rank techniques to the original weight matrices as low-rank branches. This allows the low-rank technique to better adapt to zero-shot image retrieval tasks, which are cross-modal retrieval tasks, increasing the model's robustness. During training, the original parameters of DINOv2 are frozen, and only the low-rank matrices in the domain-independent and domain-related modules are optimized. This invention introduces low-rank adaptation techniques from large models into cross-modal image retrieval tasks such as zero-shot sketch retrieval, maintaining or even exceeding the accuracy of training a complete large model while reducing the number of training parameters. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating a zero-sample cross-modal image retrieval method based on low-rank adaptation provided in an embodiment of the present invention.

[0047] Figure 2 This is a flowchart illustrating the training process of the retrieval model in an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of the retrieval model in an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of the structure of a zero-sample cross-modal image retrieval device based on low-rank adaptation provided in an embodiment of the present invention. Detailed Implementation

[0050] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0051] Among existing zero-shot image retrieval methods, knowledge distillation-based approaches are ill-suited for multimodal image retrieval tasks and suffer from low training efficiency. These methods typically utilize a pre-trained teacher model to transfer knowledge to the student model backbone. However, since the data used to train the teacher model often contains only a single modality, it cannot help the student model backbone learn discriminative information about the images to be retrieved. Furthermore, the teacher model cannot participate in image retrieval during the inference phase, resulting in poor overall training efficiency. In contrast, large-model-based methods leverage the large multimodal datasets used in pre-training, enabling them to handle cross-modal image retrieval tasks effectively without requiring additional semantic information from the teacher model. However, their massive parameter count necessitates greater hardware resources for training, leading to higher training costs and slower inference speeds.

[0052] To overcome the above problems, this invention, based on low-rank technology and combining the characteristics of images from two modal domains, designs a zero-sample cross-modal image retrieval method and apparatus based on low-rank adaptation. Please see [link to related documentation]. Figure 1 The method may include the following steps:

[0053] S1, Obtain the retrieval model;

[0054] The retrieval model is obtained by using a preset low-rank technique to replace the corresponding layers in the transformer block of the large model DINOv2 with pre-designed domain-independent and domain-dependent modules, thereby adding parallel low-rank matrix branches to the original weights. After freezing the gradients of the original network parameters, it is trained using a training set containing training samples of the first and second modalities, and verified by a test set after training. The first and second modalities are different modalities, and the categories of the samples in the test set and the training set are different.

[0055] It should be noted that in S1, the original weight matrix is ​​no longer frozen when the retrieval model is used. At this time, the model parameters include the previously frozen parameters and all low-rank matrices determined during the training phase.

[0056] S2, input the first modality image to be retrieved into the retrieval model to obtain the corresponding output vector;

[0057] S3, compare the differences between the output vector of the first modality image to be retrieved and the output vectors of the second modality image test samples of each category determined during the test set verification process, and determine a preset number of second modality image test samples that meet the low difference requirement to form a retrieval set.

[0058] The structure, processing, output vector, and process of obtaining the retrieval set through difference comparison of the retrieval model will be explained in detail later.

[0059] In this embodiment of the invention, the first modality and the second modality are different modalities, reflecting cross-modal image retrieval. The images of the first modality and the second modality in this embodiment of the invention only need to belong to different modalities; there is no specific limitation. For example, in one optional implementation, the image of the first modality can be a sketch, and the image of the second modality can be a photograph.

[0060] In this embodiment of the invention, the sample categories of the training set and the test set are completely different, so that the knowledge learned by the model during the training phase is tested on completely new categories, which embodies the idea of ​​"zero samples".

[0061] This invention utilizes a pre-designed low-rank technique to improve the existing large model DINOv2. Certain corresponding layers in its transformer block are replaced with domain-independent and domain-dependent modules pre-designed in this invention. For the replaced layers, this replacement method involves retaining their original weights and adding low-rank matrix branches constructed using the low-rank technique for parallel connection, thereby adjusting the model structure. The domain-independent and domain-dependent modules are designed considering the similarity and differences between the first modality image to be retrieved and the second modality image to be retrieved.

[0062] To facilitate understanding of the present invention, the training process of the retrieval model will first be explained. Please refer to [link to documentation]. Figure 2 The training process of the retrieval model includes the following steps:

[0063] S01, Obtain the pre-trained large model DINOv2 with network parameters;

[0064] The large model DINOv2 is an existing large model whose structure includes a patch embedding layer (Patch layer for short), N transformer blocks and a multilayer perceptron (MLP head layer) connected in sequence.

[0065] This invention can use the pre-training parameters of the already disclosed large model DINOv2 to obtain the pre-trained large model DINOv2.

[0066] S02, using the preset low-rank technique, the corresponding layers in the preset modules of the first N-1 transformer blocks of the large model DINOv2 are replaced with the domain-independent modules, the corresponding layers in the preset modules of the Nth transformer block are replaced with the domain-independent modules and the domain-related modules, and the corresponding layers in the preset modules of the Nth transformer block are replaced with the domain-related modules, thus obtaining the completed retrieval model;

[0067] The domain-independent module and the domain-dependent module are obtained by retaining the original weights in the replaced layer and adding low-rank matrix branches for parallel summation. The low-rank matrix branches are formed by concatenating low-rank matrices constructed by the preset low-rank technique.

[0068] In this embodiment of the invention, the preset low-rank technique can be a low-rank adapted LoRA technique, but it is not limited to this.

[0069] To facilitate understanding of the present invention, the low-rank adaptation technique will be briefly introduced below.

[0070] Low-rank adaptation techniques (LoRA, Hu et al. 2021) are lightweight training methods that significantly reduce the number of training parameters and training overhead, while achieving or even surpassing the accuracy of training a complete model. In its implementation, LoRA freezes the gradients of the entire model and inserts a small number of low-rank matrices as weights. The forward propagation of a conventional model can be mathematically described as follows:

[0071]

[0072] Among them, W i ∈R d×k b i ∈R k d and k are the original weights and biases of the i-th layer, respectively. Here, d and k are natural numbers.

[0073] LoRA freezes the weights W of the original model when fine-tuning a large model. i and deviation b i Two low-rank matrices A and B are introduced into the forward propagation. This can be described mathematically as follows:

[0074]

[0075] Among them, A i B i These are the low-rank matrices A and B of the i-th layer; for low-rank matrices A and B, B∈R d×r , A∈R r×k , r << min(d,k), indicating low rank. During the inference process, matrices A and B can be merged into W. i In this process, we ensure that the introduction of low-rank matrices does not add any additional delay to the inference process.

[0076] Because the weights and biases are frozen, the model training and inference processes only involve these low-rank matrices. Due to the small number of parameters in the low-rank matrices, LoRA has the advantage of low training cost. Therefore, LoRA is an efficient method for fine-tuning large models.

[0077] Based on this, the present invention can introduce low-rank adaptation technology into the field of zero-shot image retrieval and design retrieval frameworks for different domains, so that the technology can better adapt to the learning of two modalities of images. For ease of understanding, the following text uses zero-shot sketch retrieval using low-rank adaptation LoRA technology, with the image of the first modality as a sketch and the image of the second modality as a photograph as a specific example. However, it should be understood that this example does not constitute a limitation on the solution of the present invention.

[0078] When the preset low-rank technique is low-rank adapted LoRA technique, S02 may include:

[0079] For the N transformer blocks of the large model DINOv2, the low-rank adaptive LoRA technique is used to replace the query-key-value layer and projection layer in the multi-head attention module and the two linear layers in the linear module in the first N-1 transformer blocks with the domain-independent module. The query-key-value layer in the multi-head attention module in the Nth transformer block is replaced with the domain-independent module, the projection layer is replaced with the domain-related module, and the two linear layers in the linear module in the Nth transformer block are replaced with the domain-related module, thus obtaining the completed retrieval model.

[0080] In this embodiment of the invention, when LoRA technology is used, the domain-independent module is denoted as the DA-LoRA module, and the domain-dependent module is denoted as the DS-LoRA module. The low-rank matrix branch is formed by concatenating low-rank matrices A and B constructed using LoRA technology.

[0081] In the large model DINOv2, the multi-head attention module is the Attention module, the query-key-value layer is the QKV layer, the projection layer is the Projection layer, and the linear module is the MLP module. For details, please refer to the relevant technical understanding of the large model DINOv2, which will not be explained in detail here.

[0082] Please see Figure 3 Understand the structure of the completed retrieval model. The first N-1 transformer blocks are called TransformerBlocks (see [link to relevant documentation]). Figure 3 (As shown in the green box at the bottom center), the Nth transformer block is the Transformer Last Block (see [link]). Figure 3 (As shown in the green box at the top center). For simplicity, only one of the first N-1 transformer blocks is shown as an example.

[0083] As a concrete example, it can be seen that for the first N-1 transformer blocks, the QKV layer, the Projection layer, and the MLP module in the Attention module ( Figure 3 In this embodiment of the multilayer perceptron (MLP), the two linear layers (Linear1 and Linear2 in the MLP) are replaced with DA-LoRA modules. The structure of the DA-LoRA module is shown in light blue. It has two branches: the left branch is the original weight W, which is retained and can be used as the original weight branch; the right branch is a newly added low-rank matrix branch, including concatenated low-rank matrices B and A, which is constructed according to LoRA technology. These two branches are connected in parallel, and the corresponding output results are summed.

[0084] For the Nth transformer block, the QKV layer in the Attention module is replaced with the DA-LoRA module, the Projection layer in the Attention module, and the two linear layers in the MLP module (i.e., Linear1 and Linear2 in the MLP) are all replaced with the DS-LoRA module. The structure of the DS-LoRA module is shown in yellow, and it has three branches. The left side is the original weight W that is retained, which can be used as the original weight branch. The two on the right are the low-rank matrix branches newly added in this embodiment of the invention, each including a cascaded low-rank matrix B and A. This low-rank matrix branch is constructed according to the LoRA technique. Each low-rank matrix branch is connected in parallel with the original weight branch, and the corresponding output results are summed.

[0085] from Figure 3 As can be seen, each low-rank matrix branch includes a cascaded low-rank matrix B and a low-rank matrix A; however, for all DA-LoRA and DS-LoRA modules, the low-rank matrix B of different low-rank matrix branches can be different, and similarly, the low-rank matrix A of different low-rank matrix branches can be different.

[0086] The weight matrix W has a dimension of d×k; the low-rank matrices B and A have dimensions of d×r and r×k, respectively, where r is the rank. The value of r satisfies the low-rank requirement.

[0087] In this embodiment of the invention, the DA-LoRA module and the DS-LoRA module do not change the structure of the original modules in the Vision Transformer (i.e., the large model DINOv2). They only add parallel low-rank matrix branches (containing two low-rank matrices A and B) on the side. This design is independent of the specific structure of the model and has excellent portability and plug-and-play features.

[0088] The usage of the DA-LoRA and DS-LoRA modules will be explained later.

[0089] S03, freeze the gradient of the original network parameters of the large model DINOv2, and for the current round of training, input a small batch of the training set containing the first modality image training samples and the second modality image training samples into the completed retrieval model, and obtain the output vector of each training sample through the feature extraction processing of the retrieval model.

[0090] The categories of the training samples in the training set are known.

[0091] The gradients of the original network parameters in the large model DINOv2 are frozen, meaning the weight matrix W in all weight branches is frozen. The parameters to be learned are the low-rank matrices A and B in the DA-LoRA and DS-LoRA modules.

[0092] For the current round of training, a small batch of the training set containing training samples from the first modality and training samples from the second modality is input into the completed retrieval model. Through feature extraction processing by the retrieval model, the output vector of each training sample is obtained, including:

[0093] 1) After inputting the small batch of training in the current round into the retrieval model, the patch embedding layer at the input end of the large model DINOv2 is used to map each training sample into a vector with position information, so as to obtain the input vector of each training sample.

[0094] Please see Figure 3 Understanding this, the step may include:

[0095] ① The image corresponding to each input training sample is uniformly divided into K blocks using a patch embedding layer; and each block image is mapped to a D-dimensional vector.

[0096] As a specific example, for the mini-batch of the input training set, each training sample, namely the sketch training sample and the photo training sample, is an image. The Patch layer is a learnable linear projection layer that will evenly divide the image corresponding to each input training sample into K blocks. For each of the K blocks, the block image will be mapped to a D-dimensional vector.

[0097] Where K and D are natural numbers.

[0098] ② Add a learnable classification vector x class This is used as the output of the retrieval model, and an input vector is constructed for each training sample as follows:

[0099]

[0100] Among them, the learnable classification vector x class It contains initialization information during setup, x class It contains multiple elements, each corresponding to a confidence level for a given category. Through the model learning process, the final output of the model is formed.

[0101] After the images corresponding to the training samples are uniformly divided into K blocks and mapped respectively, the vector of the mapped j-th block image is D-dimensional; E pos This represents the relative position information of all elements in z0, in vector form.

[0102] 2) The input vector of each training sample is used to extract features sequentially using N transformer blocks, and the output vector of each training sample is output through the multilayer perceptron layer at the end of the large model DINOv2.

[0103] In the domain-independent module, the weight branch and the low-rank matrix branch simultaneously receive vector data from the first modality image training samples and the second modality image training samples. In the domain-related module, the weight branch is connected in parallel with two low-rank matrix branches and summed. The weight branch simultaneously receives vector data from the first modality image training samples and the second modality image training samples, while one low-rank matrix branch receives vector data from the first modality image training samples and the other low-rank matrix branch receives vector data from the second modality image training samples. The weight matrix W in all weight branches is frozen. Each low-rank matrix branch includes a concatenated low-rank matrix B and a low-rank matrix A. The dimension of the weight matrix W is d×k. The dimensions of the low-rank matrix B and the low-rank matrix A are d×r and r×k, respectively, where r is the rank, and the value of r satisfies the low-rank requirement.

[0104] The multilayer perceptron layer at the end of the large model DINOv2 is also called the MLP head layer, see [link to relevant documentation]. Figure 3 The multi-layer sensing head in the middle.

[0105] As a specific example, since sketch retrieval is a cross-modal retrieval, the model must possess a certain degree of feature robustness. This requires the model to not only understand the similar semantics (such as outlines) between sketches and photographs, but also to distinguish their respective specific semantics (such as textures). Based on this, this invention uses the DA-LoRA module and the DS-LoRA module to identify shared features and specific features of the two modalities, respectively. The DA-LoRA module aims to identify and utilize the shared features of the two modalities, thereby promoting knowledge sharing, while the DS-LoRA module aims to capture specific features of a particular domain.

[0106] Specifically, in the DA-LoRA module, the weight branch and the low-rank matrix branch simultaneously receive vector data from both sketch training samples and photograph training samples; in the DS-LoRA module, the weight branch is connected in parallel with two low-rank matrix branches and summed. The weight branch simultaneously receives vector data from both sketch training samples and photograph training samples, while one low-rank matrix branch receives vector data from sketch training samples and the other low-rank matrix branch receives vector data from photograph training samples.

[0107] Specifically, for the first N-1 transformer blocks, the output z of any transformer block l z′ l =Attention(LN(z) l-1 ))+DALoRA Attention (LN(z l-1 ))+z l-1 ,l=1,…,N-1;

[0108] z l =MLP(LN(z′) l )+DALoRA MLP (LN(z′ l ))+z′ l ), l=1,…,N-1;

[0109] Among them, DALoRA Attention This indicates the integration of the DA-LoRA module into the Attention module. MLP This indicates the integration of the DA-LoRA module into the MLP module, Attention indicates the processing of the Attention module, LN() indicates the processing of the linear layer, and MLP() indicates the processing of the MLP module.

[0110] The DA-LoRA module in this embodiment of the invention aims to learn the shared information between the sketch and photo modalities. Therefore, whether it is the Attention module and MLP module of the first N-1 transformer blocks, or the QKV (integrated DA-LoRA module) in the Attention module of the last Nth transformer block, they all receive inputs from sketch and photo vectors simultaneously.

[0111] For the last Nth transformer block, combined with Figure 3 Understanding, the model's output y is:

[0112] z' N =Attention(LN(z) N-1 ))+DSLoRA Attention (LN(z N-1 ))+z N-1 ;

[0113] z N =MLP(LN(z′) N )+DSLoRA MLP (LN(z′ N ))+z′ N );

[0114] Among them, DSLoRA Attention This indicates the integration of the DS-LoRA module into the Attention module, DSLoRA. MLP This indicates the integration of the DS-LoRA module into the MLP module.

[0115] The model's output y is the output vector. Specifically, y is processed by... Figure 3 The 768-dimensional vector output after the final MLP head layer (768 depends on the MLP head layer settings). y is a vector representing x. class The result after changes at each layer of the model.

[0116] The DS-LoRA module in this embodiment of the invention aims to learn specific knowledge of a particular domain and is mainly concentrated in the MLP module in the last transformer block. Besides both being input into the frozen weight matrix W, sketch vectors are input into low-rank matrices A1 and B1 in one low-rank matrix branch, while photograph vectors are input into low-rank matrices A2 and B2 in another low-rank matrix branch. Thus, the DS-LoRA module distinguishes between these two modalities, with samples from different modalities entering different low-rank matrices. In contrast, the DA-LoRA module accepts both sketch and photograph sample vectors simultaneously, without distinguishing between modalities.

[0117] Please refer to the original feature extraction process of the large model DINOv2 and the descriptions of the DA-LoRA and DS-LoRA modules above for a better understanding of the feature extraction process in this embodiment of the invention.

[0118] S04. For each first modality image training sample, use its output vector and the output vectors of second modality image training samples of the same and different categories to calculate the value of its loss function, and obtain the value of the total loss function of all first modality image training samples in this mini-batch.

[0119] Specifically, for each first modality image training sample, the loss function is calculated using its output vector and the output vectors of second modality image training samples of the same and different classes, expressed by the formula:

[0120]

[0121] in, Training samples for the first modality image The output vector, L i Training samples for the first modality image The value of the loss function, To and The output vector of the training samples of the second modality of the same category, To and The output vectors of training samples of second modality images of different categories, d(,) is used to calculate the Euclidean distance, and α is the margin hyperparameter;

[0122] As a specific example Training samples for sketches The output vector, L i Training samples for sketches The value of the loss function, To and The output vectors of training samples of the same category, To and The output vectors of training samples from different categories of photos. The loss function described above is a triplet loss, which aims to ensure that the Euclidean distance between the output vectors of sketches and photos of the same category is close enough, and the Euclidean distance between the output vectors of sketches and photos of different categories is far enough. α is the index for measuring "distance", which can be set as needed. In this embodiment of the invention, a selectable value of 0.2 is used.

[0123] The total loss function for all first-modality image training samples in this mini-batch is obtained by the following formula:

[0124]

[0125] Where M is the number of all first-modality image training samples in this mini-batch. In the specific example, M is the number of all sketch training samples in this mini-batch.

[0126] Through the above processing, the value of the total loss function for the current round of training can be obtained.

[0127] S05, based on the gradient of the total loss function, optimize the low-rank matrices in each domain-independent module and domain-dependent module to complete the current round of training;

[0128] As a concrete example, this step optimizes the low-rank matrices A and B in each DA-LoRA and DS-LoRA module based on the gradient of the total loss function to complete the current round of training.

[0129] The gradient of the total loss function is obtained from the values ​​of the total loss function in the current training round and the total loss function in the previous training round.

[0130] As a specific example, based on the gradient of the total loss function, the low-rank matrices A and B in each DA-LoRA and DS-LoRA module are optimized using the Adam optimizer. The specific optimization process can be understood by referring to the existing model training process, and will not be explained in detail here.

[0131] The learning rate during training can be 1e-6.

[0132] S06, repeat the training multiple times until the preset maximum number of training rounds is reached to obtain the pre-trained retrieval model.

[0133] S03 to S05 execute one round of training. If the preset maximum training rounds max_epochs are not reached, S03 to S05 continue to be executed until the preset maximum training rounds max_epochs are reached, then training stops and the trained retrieval model is obtained.

[0134] The preset maximum number of training epochs, max_epochs, can be set as needed, for example, it can be 20.

[0135] This invention explores cross-modal image retrieval tasks such as zero-shot sketch retrieval by employing low-rank techniques instead of directly training a large model. The overall model is based on the Vision Transformer architecture of the large model DINOv2. Domain-independent and domain-related modules are designed to address the similarities and uniqueness of images from two different modalities. Corresponding layers in some DINOv2 modules are replaced; specifically, a low-rank matrix constructed using low-rank techniques is added to the original weight matrix as a low-rank branch. This allows the low-rank technique to better adapt to zero-shot image retrieval tasks, which are cross-modal retrieval tasks, increasing the model's robustness. During training, the original parameters of DINOv2 are frozen, and only the low-rank matrices in the domain-independent and domain-related modules are optimized. This invention introduces low-rank adaptation techniques from large models into zero-shot image retrieval tasks, maintaining or even exceeding the accuracy of training a complete large model while reducing the number of training parameters.

[0136] In a specific example using sketch retrieval, the present invention is based on a given sketch retrieval task dataset. Where y represents the i-th sample, i This represents the corresponding category label value. E and P represent the sketch and photograph modalities, respectively. The goal of zero-shot sketch retrieval is to retrieve sketches from the source dataset D. s Train a model with excellent discriminative representation on the training set (i.e., the target dataset D) and then train the model on the target dataset D. t Evaluate it on the test set (i.e., the test set). The zero-sample setup requires D... s and D t The dataset contains completely different categories. During the inference phase, sketches serve as queries, retrieving photos belonging to the same category from a photo library (which could be a set of photos in the test set). Therefore, the goal of zero-shot sketch retrieval is to learn a cross-modal representation F that generalizes well. θ (·) is used to avoid overfitting while achieving high retrieval accuracy.

[0137] The original large model DINOv2 directly trains the weight matrix W. Taking the linear module as an example, the weight matrix W is 3072*768 in size, and r(W), i.e., the rank of the weight matrix, is very large, making full training resource-intensive. This invention freezes the weight matrix W and instead trains two low-rank matrices A and B. The rank of A is 3072*64, and the rank of B is 64*768. Both ranks A and B are relatively small. At the same time, the achieved retrieval performance is superior to directly training the weight matrix W.

[0138] The domain-independent DA-LoRA module receives data from both the sketch and photo domains simultaneously to identify and utilize shared features between the two modalities. The domain-dependent DS-LoRA module, with its two low-rank matrix branches each receiving data from one modality, captures specific features of a particular domain, ensuring that LoRA technology is better suited to downstream sketch retrieval tasks and improving retrieval accuracy.

[0139] As can be seen, for cross-modal image retrieval, the present invention can reduce model training parameters while ensuring model training accuracy.

[0140] After obtaining the pre-trained retrieval model, this invention uses a test set to verify the performance of the retrieval model. If the verification is successful, it can be used for formal retrieval.

[0141] The test set verification process includes:

[0142] 1) Input the test set into the pre-trained retrieval model to obtain the output vector of each test sample;

[0143] The test set contains test samples of a first modality image and test samples of a second modality image, and the categories are known.

[0144] 2) For any first modality image test sample, compare its output vector with the output vectors of each category of second modality image test samples to determine a preset number of second modality image test samples that meet the low difference requirement, and form a neighborhood set of the first modality image test sample; determine the number of second modality image test samples in the neighborhood set that are consistent with the category of the first modality image test sample, and thus obtain the retrieval model accuracy corresponding to the first modality image test sample;

[0145] In this step, the output vectors of the second modality image test samples of each category can be saved for use in the formal retrieval phase.

[0146] Specifically, for any first modality image test sample, the difference between its output vector and the output vectors of second modality image test samples of each category is compared to determine a preset number of second modality image test samples that meet the low difference requirement, forming a neighborhood set of the first modality image test sample, including:

[0147] For any first modality image test sample, calculate the Euclidean distance between its output vector and the output vectors of the second modality image test samples of each category. Sort them in ascending order of Euclidean distance, and take a preset number of second modality image test samples at the top of the sort to form a neighborhood set.

[0148] Similarly, in step S3 of this invention, the Euclidean distance is calculated between the output vector of the first modality image to be retrieved and the output vector of the second modality image test samples of each category. The samples are sorted in ascending order of Euclidean distance, and a preset number of second modality image test samples at the top of the sorting are taken to form a retrieval set, thereby obtaining the retrieval result. The second modality image test samples in the retrieval set are considered to be consistent with or close to the category of the first modality image to be retrieved, that is, they meet the low difference requirement.

[0149] The preset quantity can be selected as needed, such as 100 or 200.

[0150] Specifically, after determining the number of second modality image test samples in the neighborhood set that are consistent with the category of the first modality image test sample, the retrieval model accuracy corresponding to the first modality image test sample can be obtained by using the ratio of this number to a preset number.

[0151] 3) Utilize the accuracy of the retrieval model corresponding to all the first modality image test samples to obtain the final accuracy of the retrieval model, and verify that the final accuracy meets the requirements.

[0152] Specifically, the final accuracy of the retrieval model can be obtained by averaging the accuracy of the retrieval models corresponding to all the first modality image test samples.

[0153] The quantitative metrics for final accuracy can be mAP and Prec.

[0154] Understandably, if the final accuracy does not meet the requirements, the verification will fail, and the system can be retrained and verified again until the verification passes.

[0155] This invention validates the trained retrieval model again using a test set. If the accuracy requirements are met, a retrieval model with applicable performance can be obtained. If the accuracy requirements are not met, the model can be trained and adjusted again to meet the requirements and improve the accuracy of the retrieval stage.

[0156] In a specific example, embodiments of the present invention utilize a trained retrieval model to obtain multiple photos of sketches of the same or similar categories, thereby completing the sketch retrieval. This method maintains or even surpasses the accuracy of conventional methods while improving inference speed.

[0157] Based on the above method embodiments, the present invention also provides a zero-sample cross-modal image retrieval device based on low-rank adaptation. Please refer to [link to relevant documentation]. Figure 4 The device includes:

[0158] A retrieval model acquisition module is used to acquire a retrieval model. The retrieval model is obtained by using a preset low-rank technique to replace the corresponding layers in the transformer block of the large model DINOv2 with pre-designed domain-independent and domain-dependent modules, thereby adding parallel low-rank matrix branches to the original weights. After freezing the gradients of the original network parameters, the model is trained using a training set containing training samples of the first and second modalities, and validated using a test set after training. The first and second modalities are different modalities, and the categories of samples in the test set and the training set are different.

[0159] The retrieval model processing module is used to input the first modality image to be retrieved into the retrieval model and obtain the corresponding output vector;

[0160] The retrieval set determination module is used to compare the output vector of the first modality image to be retrieved with the output vector of the second modality image test samples of each category determined during the test set verification process, and determine a preset number of second modality image test samples that meet the low difference requirement to form the retrieval set.

[0161] In one optional implementation, the training process of the retrieval model includes:

[0162] Obtain the pre-trained large model DINOv2 with network parameters;

[0163] Using a pre-defined low-rank technique, the corresponding layers in the pre-defined modules within the first N-1 transformer blocks of the large model DINOv2 are replaced with the domain-independent modules. The corresponding layers in the pre-defined modules within the Nth transformer block are replaced with both the domain-independent and domain-related modules. The corresponding layers in the pre-defined modules within the Nth transformer block are replaced with the domain-related modules, resulting in a completed retrieval model. The domain-independent and domain-related modules are obtained by retaining the original weights within the replaced layers and adding low-rank matrix branches for parallel summation. These low-rank matrix branches are constructed by concatenating low-rank matrices built using the pre-defined low-rank technique.

[0164] The gradients of the original network parameters of the large model DINOv2 are frozen. For the current training round, a small batch of the training set containing the first modality image training samples and the second modality image training samples is input into the completed retrieval model. Through feature extraction processing of the retrieval model, the output vector of each training sample is obtained; wherein, the category of the training samples in the training set is known.

[0165] For each first modality image training sample, its output vector and the output vectors of second modality image training samples of the same and different classes are used to calculate the value of its loss function, and the value of the total loss function of all first modality image training samples in this mini-batch is obtained.

[0166] Based on the gradient of the total loss function, optimize the low-rank matrices in each domain-independent and domain-dependent module to complete the current round of training;

[0167] Repeat the training multiple times until the preset maximum number of training rounds is reached to obtain the pre-trained retrieval model.

[0168] In one optional implementation, when the preset low-rank technique is low-rank adapted LoRA technique, the step of using the preset low-rank technique to replace the corresponding layers in the preset modules of the first N-1 transformer blocks of the large model DINOv2 with the domain-independent modules, replacing the corresponding layers in the preset modules of the Nth transformer block with the domain-independent modules and the domain-related modules, and replacing the corresponding layers in the preset modules of the Nth transformer block with the domain-related modules, results in the completed retrieval model, including:

[0169] For the N transformer blocks of the large model DINOv2, the low-rank adaptive LoRA technique is used to replace the query-key-value layer and projection layer in the multi-head attention module and the two linear layers in the linear module in the first N-1 transformer blocks with the domain-independent module. The query-key-value layer in the multi-head attention module in the Nth transformer block is replaced with the domain-independent module, the projection layer is replaced with the domain-related module, and the two linear layers in the linear module in the Nth transformer block are replaced with the domain-related module, thus obtaining the completed retrieval model.

[0170] In one optional implementation, for the current round of training, a small batch of the training set containing training samples from the first modality and training samples from the second modality is input into the completed retrieval model. Through feature extraction processing by the retrieval model, the output vector of each training sample is obtained, including:

[0171] After inputting the small batch of training in the current round into the retrieval model, the patch embedding layer at the input end of the large model DINOv2 is used to map each training sample into a vector with position information, thus obtaining the input vector of each training sample.

[0172] The input vectors of each training sample are sequentially processed using N transformer blocks for feature extraction, and the output vectors of each training sample are output through the multilayer perceptron layer at the end of the large model DINOv2. The weight branch and low-rank matrix branch in the domain-independent module simultaneously receive vector data from the first and second modal image training samples. In the domain-related module, the weight branch is connected in parallel with two low-rank matrix branches and summed. The weight branch simultaneously receives vector data from the first and second modal image training samples, while one low-rank matrix branch receives vector data from the first modal image training sample, and the other receives vector data from the second modal image training sample. The weight matrix W in all weight branches is frozen. Each low-rank matrix branch includes a concatenated low-rank matrix B and a low-rank matrix A. The dimension of the weight matrix W is d×k. The dimensions of the low-rank matrix B and low-rank matrix A are d×r and r×k, respectively, where r is the rank, and the value of r satisfies the low-rank requirement.

[0173] In one optional implementation, the patch embedding layer at the input of the large model DINOv2 is used to map each training sample into a vector with positional information, thereby obtaining the input vector of each training sample, including:

[0174] The patch embedding layer is used to uniformly divide the image corresponding to each input training sample into K blocks; and each block image is mapped to a D-dimensional vector.

[0175] Add a learnable classification vector x class This is used as the output of the retrieval model, and an input vector is constructed for each training sample as follows:

[0176]

[0177] in, After the images corresponding to the training samples are uniformly divided into K blocks and mapped respectively, the vector of the mapped j-th block image is D-dimensional; E pos This represents the relative position information of all elements in z0, in vector form.

[0178] In one optional implementation, for each first modality image training sample, the loss function is calculated using its output vector and the output vectors of second modality image training samples of the same and different classes, expressed by the formula:

[0179]

[0180] in, Training samples for the first modality image The output vector, L i Training samples for the first modality image The value of the loss function, To and The output vector of the training samples of the second modality of the same category, To and The output vectors of training samples of second modality images of different categories, d(,) is used to calculate the Euclidean distance, and α is the margin hyperparameter;

[0181] The value of the total loss function obtained for all first-modality image training samples in this mini-batch is expressed by the formula:

[0182]

[0183] Where M is the number of all first-modality image training samples in this mini-batch.

[0184] In one optional implementation, the low-rank matrices in each domain-independent module and domain-dependent module are optimized based on the gradient of the total loss function to complete the current round of training, which is achieved using the Adam optimizer.

[0185] In one optional implementation, the test set verification process includes:

[0186] The test set is input into the pre-trained retrieval model to obtain the output vector of each test sample; wherein the test set contains first modality image test samples and second modality image test samples, and the categories are known;

[0187] For any first modality image test sample, its output vector is compared with the output vectors of second modality image test samples of each category to determine a preset number of second modality image test samples that meet the low difference requirement, which constitute the neighborhood set of the first modality image test sample; the number of second modality image test samples in the neighborhood set that are consistent with the category of the first modality image test sample is determined, thereby obtaining the retrieval model accuracy corresponding to the first modality image test sample;

[0188] The final accuracy of the retrieval model is obtained by using the accuracy of the retrieval model corresponding to all the first modality image test samples. The model is verified when the final accuracy meets the requirements.

[0189] In one optional implementation, the step of comparing the difference between the output vector of any first modality image test sample and the output vectors of second modality image test samples of each category to determine a preset number of second modality image test samples that meet the low difference requirement, thus forming a neighborhood set of the first modality image test sample, includes:

[0190] For any first modality image test sample, calculate the Euclidean distance between its output vector and the output vectors of the second modality image test samples of each category. Sort them in ascending order of Euclidean distance, and take a preset number of second modality image test samples at the top of the sort to form a neighborhood set.

[0191] For details about the device, please refer to the previous section on the zero-sample cross-modal image retrieval method based on low-rank adaptation; further explanation is not required here.

[0192] This invention introduces large model low-rank adaptation technology into zero-sample cross-modal image retrieval tasks such as zero-sample sketch retrieval, which can maintain or even exceed the accuracy of training a complete large model while reducing the number of model training parameters.

[0193] To verify the effectiveness of the embodiments of the present invention, relevant simulation experimental data are given below for the specific examples described above.

[0194] First, the performance of the self-supervised visual large model DINOv2 was evaluated on the publicly available zero-sample sketch retrieval dataset Sketchy Ext based on triplet loss.

[0195] The Sketchy Ext dataset contains 125 categories, a total of 75,471 sketches and 12,500 photographs. Liu et al. collected an additional 60,502 natural photographs in 2019, for a total of 73,002 photographs.

[0196] To verify the effectiveness of this invention, evaluation metrics from other retrieval methods were used, including mean accuracy (mAP) and precision (Prec). mAP and Prec are indicators that comprehensively evaluate the overall performance of the retrieval system; higher values ​​indicate better retrieval results. Experimental results are shown in Table 1.

[0197] Table 1. Experimental results of zero-sample sketch retrieval based on domain adaptation.

[0198]

[0199] Among them, the LoRA-only method means that only the DA-LoRA module of the present invention is used, and the DS-LoRA module is not used.

[0200] As shown in Table 1, simple LoRA (LoRA-only) achieves performance similar to fully retraining DINOv2 with very few parameters, but its overall performance is still inferior to fully training the entire model. It is worth noting that the latest zero-shot sketch retrieval methods perform at a level lower than or similar to those using the DINOv2 model. For example, in 2023, Aneeshan Sain et al. proposed the CLIP-AT retrieval method, achieving mAP@200 and Prec@200 of 0.723 and 0.725 respectively on this dataset; in 2024, Subhadeep Koley et al. proposed TTI, achieving mAP@200 and Prec@200 of 0.746 and 0.747 respectively. These methods all involve fully training a model with a large number of parameters. On the other hand, compared to fully training the DINOv2 model, the method of this invention utilizes approximately 12% of the parameters while achieving the best performance on the Sketchy Ext dataset. This further demonstrates the superiority of the method of this invention. The method of this invention surpasses both simple LoRA and fully parameter-trained strategies while reducing training parameters and time overhead.

[0201] In summary, the accuracy of the method described in this invention surpasses that of training a large model in its entirety and simply applying LoRA techniques. Furthermore, the method is not specific to any particular model architecture, making it easy to deploy in real-world scenarios.

[0202] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0203] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A zero-sample cross-modal image retrieval method based on low-rank adaptation, characterized in that, include: A retrieval model is obtained; wherein the retrieval model is obtained by using a preset low-rank technique to replace the corresponding layers in the transformer block of the large model DINOv2 with pre-designed domain-independent modules and domain-dependent modules, thereby adding parallel low-rank matrix branches to the original weights, freezing the gradients of the original network parameters, training with a training set containing training samples of the first modality and training samples of the second modality, and verifying the results with a test set after training; the first modality and the second modality are different modalities, and the categories of the samples in the test set and the training set are different; The first modality image to be retrieved is input into the retrieval model to obtain the corresponding output vector; The output vector of the first modality image to be retrieved and the output vectors of the second modality image test samples of each category determined in the test set verification process are compared for differences. A preset number of second modality image test samples that meet the low difference requirement are determined to form the retrieval set. The training process of the retrieval model includes: Obtain the pre-trained large model DINOv2 with network parameters; For the N transformer blocks of the large model DINOv2, the low-rank adaptive LoRA technique is used to replace the query-key-value layer and projection layer in the multi-head attention module and the two linear layers in the linear module in the first N-1 transformer blocks with the domain-independent module. The query-key-value layer in the multi-head attention module in the Nth transformer block is replaced with the domain-independent module, the projection layer is replaced with the domain-related module, and the two linear layers in the linear module in the Nth transformer block are replaced with the domain-related module, thus obtaining the completed retrieval model. The domain-independent module and the domain-related module are obtained by retaining the original weights in the replaced layers and adding low-rank matrix branches for parallel summation. The low-rank matrix branches are constructed by concatenating low-rank matrices built by the preset low-rank technique. The gradients of the original network parameters of the large model DINOv2 are frozen. For the current training round, a small batch of the training set containing the first modality image training samples and the second modality image training samples is input into the completed retrieval model. Through feature extraction processing of the retrieval model, the output vector of each training sample is obtained; wherein, the category of the training samples in the training set is known. For each first modality image training sample, its output vector and the output vectors of second modality image training samples of the same and different classes are used to calculate the value of its loss function, and the value of the total loss function of all first modality image training samples in this mini-batch is obtained. Based on the gradient of the total loss function, optimize the low-rank matrices in each domain-independent and domain-dependent module to complete the current round of training; Repeat the training multiple times until the preset maximum number of training rounds is reached to obtain the pre-trained retrieval model.

2. The zero-sample cross-modal image retrieval method based on low-rank adaptation according to claim 1, characterized in that, For the current round of training, a small batch of the training set containing training samples from the first modality and training samples from the second modality is input into the completed retrieval model. Through feature extraction processing by the retrieval model, the output vector of each training sample is obtained, including: After inputting the small batch of training in the current round into the retrieval model, the patch embedding layer at the input end of the large model DINOv2 is used to map each training sample into a vector with position information, thus obtaining the input vector of each training sample. The input vectors of each training sample are sequentially processed using N transformer blocks for feature extraction, and the output vectors of each training sample are output through the multilayer perceptron layer at the end of the large model DINOv2. The weight branch and low-rank matrix branch in the domain-independent module simultaneously receive vector data from both the first and second modal image training samples. In the domain-correlated module, the weight branch is connected in parallel with two low-rank matrix branches and summed. The weight branch simultaneously receives vector data from both the first and second modal image training samples, while one low-rank matrix branch receives vector data from the first modal image training sample, and the other low-rank matrix branch receives vector data from the second modal image training sample. All weight branches contain weight matrices... Frozen; each low-rank matrix branch includes a concatenated low-rank matrix B and a low-rank matrix A; weight matrix The dimension is The dimensions of low-rank matrix B and low-rank matrix A are respectively... , , For rank, The value of satisfies the low-rank requirement.

3. The zero-sample cross-modal image retrieval method based on low-rank adaptation according to claim 2, characterized in that, By using the patched embedding layer at the input of the large model DINOv2, each training sample is mapped to a vector with positional information, resulting in the input vector for each training sample, including: The patch embedding layer is used to uniformly divide the image corresponding to each input training sample into... Divide the image into blocks; and map each block image as... A dimensional vector; Add a learnable classification vector This is used as the output of the retrieval model, and an input vector is constructed for each training sample as follows: ； in, The images corresponding to the training samples are uniformly divided After dividing the blocks and mapping them separately, the first block... The vectors mapped from each image block are: dimension; express The relative position information of all elements is in vector form.

4. The zero-sample cross-modal image retrieval method based on low-rank adaptation according to claim 2, characterized in that, For each training sample of the first modality image, the loss function is calculated using its output vector and the output vectors of training samples of the same and different classes of the second modality images, expressed by the formula: ； in, Training samples for the first modality image The output vector, Training samples for the first modality image The value of the loss function, To and The output vector of the training samples of the second modality of the same category, To and Output vectors of training samples of second modality images of different categories. To calculate the Euclidean distance, For interval hyperparameters; The value of the total loss function obtained for all first-modality image training samples in this mini-batch is expressed by the formula: ； in, This represents the number of training samples for all first-modality images in this mini-batch.

5. The zero-sample cross-modal image retrieval method based on low-rank adaptation according to claim 2, characterized in that, The gradient based on the total loss function is used to optimize the low-rank matrices in each domain-independent and domain-dependent module to complete the current round of training, which is achieved using the Adam optimizer.

6. The zero-sample cross-modal image retrieval method based on low-rank adaptation according to claim 1, characterized in that, The test set validation process includes: The test set is input into the pre-trained retrieval model to obtain the output vector of each test sample; wherein the test set contains first modality image test samples and second modality image test samples, and the categories are known; For any first modality image test sample, its output vector is compared with the output vectors of second modality image test samples of each category to determine a preset number of second modality image test samples that meet the low difference requirement, which constitute the neighborhood set of the first modality image test sample; the number of second modality image test samples in the neighborhood set that are consistent with the category of the first modality image test sample is determined, thereby obtaining the retrieval model accuracy corresponding to the first modality image test sample; The final accuracy of the retrieval model is obtained by using the accuracy of the retrieval model corresponding to all the first modality image test samples. The model is verified when the final accuracy meets the requirements.

7. The zero-sample cross-modal image retrieval method based on low-rank adaptation according to claim 6, characterized in that, For any first modality image test sample, the difference between its output vector and the output vectors of second modality image test samples of each category is compared to determine a preset number of second modality image test samples that meet the low difference requirement, forming a neighborhood set of the first modality image test sample, including: For any first modality image test sample, calculate the Euclidean distance between its output vector and the output vectors of the second modality image test samples of each category. Sort them in ascending order of Euclidean distance, and take a preset number of second modality image test samples at the top of the sort to form a neighborhood set.

8. A zero-sample cross-modal image retrieval device based on low-rank adaptation, characterized in that, include: A retrieval model acquisition module is used to acquire a retrieval model. The retrieval model is obtained by using a preset low-rank technique to replace the corresponding layers in the transformer block of the large model DINOv2 with pre-designed domain-independent and domain-dependent modules, thereby adding parallel low-rank matrix branches to the original weights. After freezing the gradients of the original network parameters, the model is trained using a training set containing training samples of the first and second modalities, and validated using a test set after training. The first and second modalities are different modalities, and the categories of samples in the test set and the training set are different. The retrieval model processing module is used to input the first modality image to be retrieved into the retrieval model and obtain the corresponding output vector; The retrieval set determination module is used to compare the output vector of the first modality image to be retrieved with the output vector of the second modality image test samples of each category determined during the test set verification process, and determine a preset number of second modality image test samples that meet the low difference requirement to form the retrieval set; The training process of the retrieval model includes: Obtain the pre-trained large model DINOv2 with network parameters; For the N transformer blocks of the large model DINOv2, the low-rank adaptive LoRA technique is used to replace the query-key-value layer and projection layer in the multi-head attention module and the two linear layers in the linear module in the first N-1 transformer blocks with the domain-independent module. The query-key-value layer in the multi-head attention module in the Nth transformer block is replaced with the domain-independent module, the projection layer is replaced with the domain-related module, and the two linear layers in the linear module in the Nth transformer block are replaced with the domain-related module, thus obtaining the completed retrieval model. The domain-independent module and the domain-related module are obtained by retaining the original weights in the replaced layers and adding low-rank matrix branches for parallel summation. The low-rank matrix branches are constructed by concatenating low-rank matrices built by the preset low-rank technique. The gradients of the original network parameters of the large model DINOv2 are frozen. For the current training round, a small batch of the training set containing the first modality image training samples and the second modality image training samples is input into the completed retrieval model. Through feature extraction processing of the retrieval model, the output vector of each training sample is obtained; wherein, the category of the training samples in the training set is known. For each first modality image training sample, its output vector and the output vectors of second modality image training samples of the same and different classes are used to calculate the value of its loss function, and the value of the total loss function of all first modality image training samples in this mini-batch is obtained. Based on the gradient of the total loss function, optimize the low-rank matrices in each domain-independent and domain-dependent module to complete the current round of training; Repeat the training multiple times until the preset maximum number of training rounds is reached to obtain the pre-trained retrieval model.

Images