A construction method based on a deep fusion model and cross-modal data hash retrieval

Abstract

The application discloses a construction method based on a deep fusion model and cross-modal data hash retrieval, and is committed to simultaneously compatible image retrieval and cross-modal retrieval technology; through integration of joint training, knowledge distillation and deep hash retrieval method, the architecture of the application can effectively process diversified image and text data, while improving retrieval efficiency and accuracy; solves the problems of large scale, high interaction complexity and low retrieval efficiency of traditional models.

Description

Technical Field

[0001] This invention belongs to the field of cross-modal retrieval technology, specifically relating to a construction method based on deep fusion model and cross-modal data hash retrieval. Background Technology

[0002] In the current technological context, large single-modal models such as GPT (text) and ResNet (image) have achieved significant progress in their respective fields. However, they are limited to processing single-modal data and have limited capabilities when facing cross-modal retrieval tasks. On the other hand, large cross-modal models such as CLIP and ViLBERT improve the performance of cross-modal retrieval by jointly understanding image and text information. However, these models often fail to deeply mine the rich information within a single modality, resulting in poor performance on retrieval tasks within a specific modality. This leads to unsatisfactory performance on specific modal retrieval tasks, such as image retrieval alone.

[0003] In addition, researchers have proposed many learning-based hashing methods for image retrieval, aiming to map similar images to close binary hash codes in Hamming space through compact, similarity-preserving representations. These supervised hashing learning methods have achieved remarkable results with the support of manually labeled images.

[0004] After investigation, the applicant found the following problems with the existing technology:

[0005] 1) While large single-modal models have made progress in their respective fields, they have limitations when handling cross-modal tasks. In contrast, cross-modal models improve cross-modal retrieval performance by integrating image and text information, but they are insufficient in mining deep information of a single modality.

[0006] 2) On the other hand, existing cross-modal models only utilize features at a shallow level of interaction and do not involve deeper semantic fusion.

[0007] 3) Typical pre-trained models are huge in scale and consume a lot of computing resources. Knowledge distillation can be used to distill the knowledge of existing large models into smaller, faster models for retrieval tasks.

[0008] 4) Regarding the retrieval problem for downstream tasks, existing cross-media retrieval models are slow and struggle to achieve efficient retrieval of large-scale cross-media data. For example, current mainstream deep learning-based retrieval methods mainly focus on improving retrieval accuracy, but often neglect the issue of retrieval efficiency. Summary of the Invention

[0009] In view of the defects or deficiencies of the existing technology, this application provides a construction method based on deep fusion model and cross-modal data hash retrieval.

[0010] To achieve the above objectives, the present invention adopts the following solution: a construction method based on a deep fusion model and cross-modal data hash retrieval, comprising the following steps:

[0011] Step 1: Select a multimodal model, which is used to extract high-level features from images and text;

[0012] Step 2: Add a Transformer fusion layer to the selected multimodal model to obtain a fused cross-modal model;

[0013] Step 3: Perform joint training on the fused cross-modal model based on image retrieval and cross-modal retrieval to obtain the fused model;

[0014] Step 4: Distill the knowledge of the fusion model into a lightweight dual encoder model through knowledge distillation;

[0015] Step 5: Map the image features and text features obtained from the lightweight dual encoder model into compact binary hash codes.

[0016] This invention primarily addresses the issue that most basic models require large amounts of pre-trained data, and that existing models are not well compatible with both unimodal and multimodal tasks. This invention designs a general model that deeply explores and utilizes the fine-grained relationships between text and images, mainly targeting downstream tasks such as image retrieval and cross-modal retrieval, thereby improving the applicability of multimodal models.

[0017] We designed a faster and lighter dual encoder model and used a distillation learning mechanism to use a pre-trained fusion CLIP multimodal model as a teacher model to distill knowledge into the dual encoder model. Through the dual encoder model, we can achieve fast and accurate encoding of the data to be retrieved.

[0018] This method utilizes deep hashing retrieval to perform unified indexing of cross-media data, enabling efficient retrieval. It forces the values ​​in the visual or linguistic representations obtained by the model to be compressed to between 0 and 1. It calculates the XOR operation between the query hash code and all hash codes in the feature library and sums them to obtain the Hamming distance. The Hamming distances are then sorted from smallest to largest, and the top-ranked retrieval results are returned to the user.

[0019] The method for constructing a general deep fusion model and applying knowledge distillation to optimize cross-modal data hash retrieval, as described in this invention, has the following technical advantages compared to existing technologies:

[0020] (1) Feature learning, hash learning, knowledge distillation, and cross-modal feature fusion are integrated into an architecture for a hash retrieval method for large-scale cross-modal data;

[0021] (2) Abundant training data: By combining the Flickr25k and Flickr30k datasets to train the model simultaneously, the model can learn both intermodal relationships and intramodal relationships.

[0022] (3) To address the shallow interaction between language and vision in the model, a decoder from Transformer is introduced to fuse the features of the two more deeply, effectively integrating the high-level image and text feature representations output from the CLIP multimodal model.

[0023] (4) A knowledge distillation mechanism is adopted to distill the knowledge of the fusion cross-modal model into a lighter dual encoder model;

[0024] (5) By applying hash quantization loss, the features output by the distilled lightweight dual encoder model are mapped to Hamming space to learn more compact hash codes.

[0025] (6) The evaluation results of the model show that the evaluation metrics can achieve good results on both the Flickr25k and Flickr30k datasets. Attached Figure Description

[0026] Figure 1 This is an overall structural block diagram of the construction method of the present invention based on deep fusion model and cross-modal data hash retrieval. Detailed Implementation

[0027] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0028] In the attached diagram, Sigmod represents the Sigmod activation function; Add & Norm represent layer normalization.

[0029] like Figure 1 As shown, this invention provides a construction method based on deep fusion model and cross-modal data hash retrieval, including the following steps:

[0030] Step 1: Select a multimodal model. The multimodal model is used to extract high-level features of images and text. Considering model performance, adaptability, and processing power, on the one hand, the CLIP multimodal model, with its dual-tower architecture, is pre-trained on large-scale image-text pairs, enabling it to accurately match images and text in cross-modal retrieval tasks and demonstrating high accuracy. On the other hand, the CLIP multimodal model can effectively handle different data types and is suitable for diverse tasks.

[0031] Choose appropriate pre-trained weights. For tasks that require a more nuanced understanding of images and text, you can choose a model version pre-trained on a larger dataset. The imported pre-trained weights are ViT-B / 32, which is the base version of the CLIP multimodal model used in the pre-training stage, where the image is segmented into 32x32 pixel blocks.

[0032] Specifically, given an image Reshape it into a 2D tile with a size of Where H represents the height of each image, W represents the width of each image, C represents the number of channels in the image, (P×P) is the size of each image block, and N = HW / P 2 N can also be used as the effective input sequence length of the Transformer to represent the number of image patches obtained. This processing method is a ready-made convolution, using a sliding window mechanism. Only an appropriate stride needs to be set to achieve image patching. Specifically, the convolution kernel and stride are both set to 16×16, so that the convolution operation can extract features every 16 pixels, and there is no overlap between adjacent regions. When the input image size is 224×224×3, a feature layer with a size of 14×14×768 is obtained. The height and width dimensions of the feature layer are flattened into a one-dimensional sequence to realize the transformation of the 14×14×768 feature layer into a sequence of size 196×768. A CLS Token is introduced into the sequence as a reference unit for serialized feature extraction. The result is a feature sequence of size 197×768. After the extraction of the 197×768 dimensional sequence information is completed, the sequence is fed into the Transformer encoder for further feature extraction.

[0033] Given a text dataset: S = (t1, t2, ..., t n It consists of n sub-words, t i Let each subword represent a word, and assuming that each subword exists independently, then the language model likelihood value of sentence S is equivalent to the product of the probabilities of all subwords:

[0034]

[0035] Suppose we merge two adjacent words x and y, and denote the resulting word as z. The change in the likelihood value of sentence S can then be expressed as:

[0036]

[0037] In the formula above, the change in likelihood value is the mutual information between two words. WordPiece selects two words to merge each time, and they have the largest mutual information value, which means that the two words have a strong correlation in the language model and they often appear in the corpus in an adjacent manner.

[0038] Each word is then normalized and padded, and converted into an embedding vector. These feature vectors are then processed by a Transformer layer to finally generate a feature vector representing the entire text input.

[0039] Step 2: Add a Transformer fusion layer to the selected multimodal model to obtain a fused cross-modal model; including:

[0040] First, the visual feature vector is obtained from the image feature encoding module. The text feature vector is obtained from the text feature encoding module. And projected onto the same dimension d, where K and M are the number of spatial feature vectors and labels, and the text feature vector T is projected onto the same dimension d. i As a query input to the Transformer decoder, visual features are transformed into V... i As a key and value input decoder, it performs cross-attention calculation of image spatial features and text word features through a multi-layer Transformer decoder;

[0041] The standard Transformer decoder architecture includes a self-attention module, a cross-attention module, and a feedforward network (FFN); the decoder in this invention can be represented as:

[0042]

[0043]

[0044]

[0045] Equation (1) represents the self-attention module of the Transformer decoder, and Equations (2) and (3) represent the cross-attention module and the feedforward network, respectively; in Equation (1), Represents the output of the i-th layer. Representing the output of the previous layer, Mask_SelfAttention(,,) is a masking operation used to mask future information during self-attention computation; in equation (2), Represents the output of the current layer, MultiHead represents the multi-head attention mechanism; in Equation (3), FFN is a positionally fully connected feedforward network, which is the fully connected layer in the attached figure;

[0046] In the cross-attention module, the text feature vector T output by the CLIP model... i As a query, the visual feature vector V output from the CLIP model is used. i Each text feature vector T is implemented using keys and values. i For all visual feature vectors V i The process involves in-depth querying, which selects image feature vectors with high relevance for synthesis. This allows each text vector to not only update itself but also update the query layer by layer through self-attention and cross-attention modules in each layer of the decoder. This process integrates the visual information of the image and the semantic information of the text, achieving deep cross-modal fusion of image spatial features and text information. During this process, the feature vectors continuously update themselves, and with the help of multi-head attention mechanisms and feedforward networks, information from different modalities is efficiently integrated.

[0047] fused feature vectors This represents the aggregated vector associated with each text and image feature; the fused feature vector H i Unlike the eigenvector z i In fact, it is the feature vector H output by the fused CLIP multimodal model. i Where M is the number of vectors and d is the dimension of the vectors. The formulas for semantic attention and visual attention can be obtained through calculation as follows:

[0048]

[0049]

[0050] Where tanh represents the hyperbolic tangent activation function, which maps input values ​​to between -1 and 1; softmax represents the activation function, which is typically used to transform a vector into a probability distribution such that each element in the vector is between 0 and 1, and the sum of all elements is 1. It is a calculation result based on the fusion of feature vectors and original spatial features of the image; These are trainable mapping parameters used to adjust the weights of the attention mechanism.

[0051] Step 3: Perform joint training on the fused cross-modal model based on image retrieval and cross-modal retrieval to obtain the fused model; including:

[0052] The image retrieval dataset Flickr25k and the cross-modal retrieval dataset Flick30k were selected;

[0053] An alternating sampling strategy is adopted, in which the fused cross-modal model is trained alternately in the image retrieval dataset and the cross-modal retrieval dataset. The alternating training means that in each training iteration, a batch of sample data is selected from one of the datasets, and different tasks are applied according to the data type: for the image retrieval task, a batch of data from the Flick25k dataset is extracted for training; for the cross-modal retrieval task, a batch of data from the Flick30k dataset is extracted for training.

[0054] After extracting different datasets, a global contrastive loss is selected to train the fusion model. The specific global contrastive loss is as follows:

[0055] Before processing image and text features, the length of each feature vector is adjusted by dividing it by its Euclidean norm; the Euclidean norm normalization operation is applied to each dimension of the feature vector X, x1, x2, ..., x... n Dividing each by ||x||2 yields a new vector, namely:

[0056]

[0057] The dot product is used to measure the similarity between normalized feature vectors, by calculating the visual feature vector V. i and text feature vector T i The dot product of these elements yields a similarity matrix, where each element represents the similarity between an image feature and a text feature.

[0058]

[0059] Where ||x||2 represents the Euclidean norm, used to obtain the similarity matrix sam_mat, where each element represents the similarity between an image feature and a text feature;

[0060] The cross-entropy loss function is used to quantify the difference between the model's predictions and the actual labels; the cross-entropy loss function includes an image loss function. and text loss function The specific function is represented as follows:

[0061]

[0062] Where sam_mat(i) is the similarity matrix from the i-th image to the text, and GT(i) is the ground truth index of the i-th image, where the ground truth index is the correct image-text pairing index;

[0063] Text loss Calculate the text-based loss using cross-entropy loss:

[0064]

[0065] Where sam_mat(i) is the similarity matrix from the i-th image to the text. T denoted as sam_mat(i), is the text-to-image similarity matrix; GT(i) is the ground truth index of the i-th text image, where the ground truth index is the correct text-image pairing index;

[0066] The total loss is calculated by combining image loss and text loss.

[0067]

[0068] The total loss This is the global contrast loss;

[0069] The purpose of this is to balance the losses from two different modalities (images and text), ensuring that the model gives equal attention to both types of data during the learning process. Essentially, it evaluates how the model maps images and text to a common embedding space and ensures that these embeddings can effectively reflect the matching relationships between images and text.

[0070] Step 4: Distill the knowledge from the fusion model into a lightweight dual-encoder model through knowledge distillation; including:

[0071] This invention uses a fast dual encoder as a student model for subsequent work. The dual encoder model refers to a model that uses the same backbone network as the CLIP multimodal model and achieves parameter alignment.

[0072] Representation of the input portion of the dual encoder model: Given an image and text, this invention encodes the pair into vector representations of the image and text; these representations are input to learn contextual representations and align image and text feature vectors; for the input image description, based on the work of the visual Transformer, the two-dimensional image v∈R H×W×C It will be segmented and reshaped into N image patches, where N = HW / P 2 Then v can be expressed as:

[0073]

[0074] Where C is the number of channels, (H×W) is the resolution of the input image (height and width), and (P,P) is the resolution of the image patch. The image patch is flattened into a vector and linearly projected to obtain the patch embedding. A learnable special label, denoted as [I_S], is added to the sequence. Finally, the input representation of the image is obtained by summing the patch embedding V. pos =R (N+1)×D Learnable one-dimensional position embedding and image type embedding (V) type =R D :

[0075]

[0076] in, It is the final image input representation. Linear projection

[0077] Regarding the text representation, following BERT's work, the text is marked as sub-word units using WordPiece; similarly, a sequence start token [T_S] is added to the text sequence; the text input representation... This involves embedding words and text positions using appropriate methods. pos and text type embedding T type This is used for summation; the expression is as follows:

[0078]

[0079] Where M is the number of words; after understanding the model architecture of the dual encoder, in the process of knowledge distillation from the fusion CLIP multimodal model to the fast dual encoder model, a strategy combining knowledge distillation divergence (KL) loss and contrastive loss was adopted.

[0080] The key feature representations of the pre-trained CLIP multimodal fusion model were extracted and used as the teacher model, and the fusion model was set as the prediction mode. The dual encoder network was set as the student model and used as the training mode. During the distillation process, the focus was on transferring this knowledge to the dual encoder model, with particular attention to its image and text encoding capabilities.

[0081] First, based on step 2, the output features of the teacher model are obtained: first, the image space feature vector is obtained.

[0082]

[0083] and the corresponding text label feature vector

[0084]

[0085] Where K represents the number of image spatial feature vectors, and M represents the number of text label feature vectors; since the original output generated by the fusion model is mainly a high-dimensional feature representation of the image and text; these features are used internally by the fusion model to calculate the similarity between the image and the text:

[0086]

[0087] To distill this knowledge, the similarity scores are transformed into a soft label form, i.e., a predicted probability distribution; this is achieved by applying a softmax function to a set of image and text pairs to obtain the probability that each text description matches a particular image.

[0088] In the knowledge distillation process, for the student model (dual encoder model), the distillation loss and the student model's own loss can be combined as the final objective function; KL divergence loss is used as the distillation loss, and contrastive loss is used as the student model's own loss; the following are specific application instructions for these loss functions:

[0089] Distillation Loss: KL divergence loss

[0090] KL divergence loss is used to ensure that the fused CLIP maintains a similar distribution to the dual encoder; specifically, the distribution is first obtained from the fusion model... Similarity distribution:

[0091]

[0092] in, It is the generated visual representation. It is the generated text representation. The cosine similarity between visual and textual representations is given by τ, which is a temperature parameter controlling the smoothness of the distribution. Similarly, this formula can be used to calculate similar distributions for dual-encoder networks. KL divergence loss is used to align the distribution of the trained fusion model with the distribution of the dual encoder model:

[0093]

[0094] Student model's own loss: contrastive loss

[0095] For multimodal learning tasks, such as image-text matching, contrastive loss can help student models learn how to correctly match images with relevant text descriptions; the implementation of contrastive loss has been described in detail above.

[0096] The final loss function can be a weighted sum of these two losses (KL loss and contrastive loss):

[0097]

[0098] α and β are hyperparameters used to control the contribution of the two types of losses to the total loss.

[0099] Step 5: Map the image features and text features obtained from the lightweight dual encoder model into compact binary hash codes; including:

[0100] The visual feature vector V output by the dual encoder model i Or text feature vector T i Here, it is uniformly represented as the feature vector z. i This is mapped to an n-bit Hamming space; implemented through a fully connected layer, where the weight matrix W and bias b are... i , the feature vector z i Convert to real-valued hash code; the specific formula is as follows:

[0101] h i =σ(W T z i +b i (9)

[0102] Among them, h i Let be the real-valued hash code of the i-th sample, σ be the sigmoid activation function that maps the input value to the interval (0,1), and b be the real-valued hash code of the i-th sample. i The generated binary hash code; binary quantization loss was used to convert b i The value in the input is pushed towards 0 or 1, specifically:

[0103]

[0104] Therefore, the total loss during distillation can be defined as:

[0105]

[0106] A comprehensive performance evaluation of the fusion model was conducted, including accuracy and retrieval speed. Following the MAP evaluation metric for image retrieval and the Rcall@K convention for cross-modal image retrieval, the performance of the initial CLIP multimodal model and the jointly trained fusion model were evaluated. The experimental results are shown in Table 1 below.

[0107] Table 1. Experimental results of the fusion model on the two datasets.

[0108]

[0109] This fusion model can perform image retrieval well beyond the original task of cross-modal feature image retrieval. As shown in Table 1, the impact of adding the Transformer fusion layer on the model is as follows: By acquiring the high-level feature representation of the CLIP multimodal model output and changing the input part of the multi-head attention in the Decoder of the Transformer model, deep interaction and fusion of images and text are achieved. Especially when facing cross-modal retrieval tasks, this allows the model to not only process data from a single modality but also more effectively integrate information from different sources, thereby improving the model's recognition and matching capabilities. Specifically, by adding the Transformer fusion layer, the model can establish deeper and more complex connections between different modalities. For example, when the image encoder and text encoder operate independently, they each process the input data and extract feature vectors. When these feature vectors pass through the Transformer fusion layer, the model can perform comprehensive analysis of the features and use the self-attention mechanism to explore the intrinsic connections between images and text.

[0110] Table 1 shows that after introducing the Transformer decoder into the CLIP multimodal model, the Recall@10 metric on the Flickr30k dataset is 3.2% higher than before, and the MAP metric on the Flickr25k dataset is 1.9% higher. This interaction is not limited to surface-level matching but can capture more subtle semantic differences, thereby enhancing the model's understanding capabilities.

[0111] The above description is merely a few embodiments of this application and is not intended to limit this application in any way. Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any changes or modifications made by those skilled in the art without departing from the scope of the technical solution of this application using the disclosed technical content are equivalent to equivalent implementation cases and fall within the scope of the technical solution.

Claims

1. A construction method based on deep fusion model and cross-modal data hash retrieval, characterized in that, Includes the following steps: Step 1: Select a multimodal model, which is used to extract high-level features from images and text; Step 2: Add a Transformer fusion layer to the selected multimodal model to obtain a fused cross-modal model; Step 3: Perform joint training on the fused cross-modal model based on image retrieval and cross-modal retrieval to obtain the fused model; Step 4: Distill the knowledge of the fusion model into a lightweight dual encoder model through knowledge distillation; Step 5: Map the image features and text features obtained from the lightweight dual encoder model into compact binary hash codes; The Transformer fusion layer includes a Transformer decoder; The text feature vector obtained by processing the multimodal model The visual feature vector is fed into the Transformer decoder as the query input. As a key-value input decoder, it performs cross-attention calculation of image spatial features and text features through the Transformer decoder; The Transformer decoder is represented as follows: ； Equation (1) represents the self-attention module of the Transformer decoder, and Equations (2) and (3) represent the cross-attention module and the feedforward network, respectively; in Equation (1), Representing the The output of the layer, Represents the output of the previous layer. It is a masking operation used to mask future information in self-attention computation; in equation (2), Represents the output of the current layer. This represents the multi-head attention mechanism; in equation (3), It is a positionally fully connected feedforward network containing two linear transformations and one activation function; Step 3 includes: The image retrieval dataset Flickr25k and the cross-modal retrieval dataset Flick30k were selected; First, the Flickr25k and Flick30k datasets are preprocessed. An alternating sampling strategy is employed, where the fused cross-modal model is trained alternately on both the image retrieval dataset and the cross-modal retrieval dataset. This alternating training means that in each training iteration, a batch of sample data is selected from one of the datasets. Different tasks are applied based on the data type: for image retrieval tasks, a batch of data from the Flick25k dataset is used for training; for cross-modal retrieval tasks, a batch of data from the Flick30k dataset is used for training. In step 4, knowledge distillation is performed to distill the knowledge of the fusion model into a lightweight dual encoder model, specifically including: The lightweight dual encoder model refers to a model that uses the same backbone network as the multimodal model and achieves parameter alignment. The fusion model obtained through joint training is used as the teacher model, and the lightweight dual encoder model is used as the student model. KL divergence loss is used as the loss function in the knowledge distillation process to measure the difference between the output of the student model and the output of the teacher model, so that the output of the student model is close to the output of the teacher model.

2. The construction method based on deep fusion model and cross-modal data hash retrieval according to claim 1, characterized in that: In step 1, the multimodal model is the CLIP multimodal model.

3. The construction method based on deep fusion model and cross-modal data hash retrieval according to claim 1, characterized in that: The high-level features of the images and text refer to the high-dimensional features obtained by the image feature encoder and text feature encoder of the multimodal model.

4. The construction method based on deep fusion model and cross-modal data hash retrieval according to claim 1, characterized in that: Before adding the Transformer fusion layer to the multimodal model, the method also includes loading pre-trained weights into the multimodal model. The pre-trained weights are ViT-B / 32.

5. The construction method based on deep fusion model and cross-modal data hash retrieval according to claim 1, characterized in that: After determining the dataset, global contrastive loss is selected to train the fusion model; The global comparison loss is specifically as follows: Before processing image and text features, the length of each feature vector is adjusted by dividing it by its Euclidean norm; the Euclidean norm normalization operation is applied to each dimension of the feature vector X. Divide by We obtain a new vector, namely: ； The dot product is used to measure the similarity between normalized feature vectors, and visual feature vectors are calculated. and text feature vectors The dot product of the two elements yields a similarity matrix, where each element represents the similarity between an image feature and a text feature. ； in The Euclidean norm is used to obtain the similarity matrix. Each element in this matrix represents the similarity between an image feature and a text feature; The cross-entropy loss function is used to quantify the difference between the model's predictions and the actual labels; the cross-entropy loss function includes an image loss function. and text loss function The specific function is represented as follows: ； in It is the first The image-to-text similarity matrix, It is the first Zhang Image index; Text loss The text-based loss is calculated using cross-entropy loss: ； in It is the first The image-to-text similarity matrix, represent The transpose of is the text-to-image similarity matrix; It is the first Zhang Wenben index; The total loss is calculated by combining image loss and text loss. : ； The total loss This is the global comparison loss.

6. The construction method based on deep fusion model and cross-modal data hash retrieval according to claim 1, characterized in that: Step 5 includes: The visual feature vector output by the dual encoder model or text feature vector They are uniformly represented as feature vectors. The feature vector is processed using a fully connected layer. Mapped to The eigenvectors are defined in a 1-bit Hamming space. The specific formula for converting to real-valued hash code is as follows: ； in, It is the first The real-valued hash code of each sample; yes An activation function that maps input values ​​to the interval (0,1); It is the weight matrix of the fully connected layer; It is the bias of the fully connected layer.

Images