Cross-modal retrieval method and device, electronic equipment and storage medium

Abstract

The invention is applicable to the technical field of artificial intelligence, and provides a cross-modal retrieval method and device, electronic equipment and a storage medium, and the method comprises the following steps: encoding an image modal through an image encoder, encoding a text modal through a text encoder, and generating a vector representation of a high-dimensional space; inputting the vector of the image mode and the vector of the text mode into the same shared self-attention layer for alignment, and extracting features through respective feedforward network layers; performing cross-modal cross fusion processing on the vector of the image modal and the vector of the text modal through a cross attention mechanism to obtain a cross-modal representation vector of the image modal and a cross-modal representation vector of the text modal; and based on the cross-modal representation vector of the image modality and the cross-modal representation vector of the text modality, outputting the similarity of the image modality and the text modality. According to the method, the cross-modal retrieval precision can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a cross-modal retrieval method, device, electronic device and storage medium. Background Art

[0002] With the explosive growth of Internet information, users need to quickly and accurately retrieve relevant information from various forms of data (such as text, images, videos, etc.). Traditional single-modal retrieval technology is limited to retrieving from the same type of data, such as keyword retrieval based only on text. Cross-modal retrieval aims to retrieve relevant data from one modality (such as text) in another modality (such as images), thereby improving the efficiency and breadth of information acquisition.

[0003] In existing multimodal tasks, the modal representations of images and texts are often not effectively integrated. Although existing methods can capture the features of images and texts to a certain extent, they show low accuracy when dealing with complex cross-modal tasks and cannot effectively complete the tasks. Summary of the invention

[0004] In view of this, embodiments of the present invention provide a cross-modal retrieval method, device, electronic device, and storage medium to improve the accuracy of cross-modal retrieval.

[0005] A first aspect of an embodiment of the present invention provides a cross-modal retrieval method, including:

[0006] Encode the image modality through the image encoder and encode the text modality through the text encoder to generate a vector representation in a high-dimensional space;

[0007] Inputting the vector of the image modality and the vector of the text modality into the same shared self-attention layer for alignment, and extracting features through their respective feed-forward network layers;

[0008] Through a cross-attention mechanism, a cross-modal cross-fusion process is performed on the vector of the image modality and the vector of the text modality to obtain a cross-modal representation vector of the image modality and a cross-modal representation vector of the text modality;

[0009] Based on the cross-modal representation vector of the image modality and the cross-modal representation vector of the text modality, the similarity between the image modality and the text modality is output.

[0010] In combination with the first aspect, in a possible implementation of the first aspect, the cross-attention mechanism is used to perform cross-modal cross-fusion processing on the vector of the image modality and the vector of the text modality to obtain the cross-modal representation vector of the image modality and the cross-modal representation vector of the text modality, including:

[0011] Inputting the vector of the image modality into an image self-attention module, and inputting the vector of the text modality into a text self-attention module;

[0012] Inputting the outputs of the image self-attention module and the text self-attention module into the image cross-attention module to obtain a cross-modal representation vector of the image modality;

[0013] The outputs of the image self-attention module and the text self-attention module are input into the text cross-attention module to obtain a cross-modal representation vector of the text modality.

[0014] In combination with the first aspect, in a possible implementation of the first aspect, the cross-modal retrieval method further includes: a training process for a cross-modal retrieval model;

[0015] The training process of the cross-modal retrieval model includes:

[0016] Two single-modal pre-trained models are introduced as teacher models for image modality and text modality respectively;

[0017] Extracting image features from the image modalities of the training set according to the teacher model of the image modality;

[0018] Extracting text features from the text modality of the training set according to the teacher model of the text modality;

[0019] Calculate similarity distribution soft labels of image modality and text modality in a training set according to the image features and the text features;

[0020] The cross-modal retrieval model is trained according to the similarity distribution soft labels.

[0021] In combination with the first aspect, in a possible implementation manner of the first aspect, training the cross-modal retrieval model according to the similarity distribution soft labels includes:

[0022] Calculating a loss function according to the similarity distribution soft labels and the similarity distribution output by the cross-modal retrieval model according to the training set;

[0023] According to the loss function, adjusting model parameters of the cross-modal retrieval model;

[0024] Continue to iterate until the preset iteration termination condition is met.

[0025] In combination with the first aspect, in a possible implementation manner of the first aspect, the loss function is a KL divergence function:

[0026]

[0027] Among them, L CSA is the KL divergence, D KL is the preset coefficient, P v2t and P t2v are respectively the similarity distribution from image modality to text modality and the similarity distribution from text modality to image modality in the similarity distribution soft label, Q v2t and Q t2v They are respectively the similarity distribution from image modality to text modality and the similarity distribution from text modality to image modality output by the cross-modal retrieval model according to the training set.

[0028] In combination with the first aspect, in a possible implementation manner of the first aspect, the image encoder is a CLIP model, and the text encoder is a RoBERTa model.

[0029] In combination with the first aspect, in a possible implementation of the first aspect, the teacher model of the image modality is a Unicom model, and the teacher model of the text modality is a Sentence-BERT model.

[0030] A second aspect of an embodiment of the present invention provides a cross-modal retrieval device, including:

[0031] An encoding module, used for encoding the image modality through an image encoder and encoding the text modality through a text encoder to generate a vector representation in a high-dimensional space;

[0032] An alignment module, used for inputting the vector of the image modality and the vector of the text modality into the same shared self-attention layer for alignment, and extracting features through their respective feedforward network layers;

[0033] A cross module, used to perform cross-modal cross-fusion processing on the vector of the image modality and the vector of the text modality through a cross-attention mechanism to obtain a cross-modal representation vector of the image modality and a cross-modal representation vector of the text modality;

[0034] An output module is used to output the similarity between the image modality and the text modality based on the cross-modal representation vector of the image modality and the cross-modal representation vector of the text modality.

[0035] In conjunction with the second aspect, in a possible implementation manner of the second aspect, the cross-connect module is used to:

[0036] Inputting the vector of the image modality into an image self-attention module, and inputting the vector of the text modality into a text self-attention module;

[0037] Inputting the outputs of the image self-attention module and the text self-attention module into the image cross-attention module to obtain a cross-modal representation vector of the image modality;

[0038] The outputs of the image self-attention module and the text self-attention module are input into the text cross-attention module to obtain a cross-modal representation vector of the text modality.

[0039] In conjunction with the second aspect, in a possible implementation of the second aspect, the cross-modal retrieval device further includes: a training module;

[0040] The training module is used to:

[0041] Two single-modal pre-trained models are introduced as teacher models for image modality and text modality respectively;

[0042] Extracting image features from the image modalities of the training set according to the teacher model of the image modality;

[0043] Extracting text features from the text modality of the training set according to the teacher model of the text modality;

[0044] Calculate similarity distribution soft labels of image modality and text modality in a training set according to the image features and the text features;

[0045] The cross-modal retrieval model is trained based on the similarity distribution soft labels.

[0046] In conjunction with the second aspect, in a possible implementation manner of the second aspect, the training module is used to:

[0047] Calculating a loss function according to the similarity distribution soft labels and the similarity distribution output by the cross-modal retrieval model according to the training set;

[0048] According to the loss function, adjusting model parameters of the cross-modal retrieval model;

[0049] Continue to iterate until the preset iteration termination condition is met.

[0050] In combination with the second aspect, in a possible implementation manner of the second aspect, the loss function is a KL divergence function:

[0051]

[0052] Among them, L CSA is the KL divergence, D KL is the preset coefficient, Pv2t and P t2v are respectively the similarity distribution from image modality to text modality and the similarity distribution from text modality to image modality in the similarity distribution soft label, Q v2t and Q t2v They are respectively the similarity distribution from image modality to text modality and the similarity distribution from text modality to image modality output by the cross-modal retrieval model according to the training set.

[0053] In combination with the second aspect, in a possible implementation manner of the second aspect, the image encoder is a CLIP model, and the text encoder is a RoBERTa model.

[0054] In combination with the second aspect, in a possible implementation of the second aspect, the teacher model of the image modality is the Unicom model, and the teacher model of the text modality is the Sentence-BERT model.

[0055] A third aspect of an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the processor implements the steps of the method in the first aspect or any one of the implementations of the first aspect.

[0056] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the method in the above-mentioned first aspect or any one of the implementation methods of the first aspect.

[0057] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0058] The embodiment of the present invention encodes the image modality through an image encoder, encodes the text modality through a text encoder, and uses a shared self-attention mechanism to achieve alignment of high-dimensional vector representations between different modalities, which can fully extract and fuse deep information of different modalities. At the same time, the cross-attention mechanism is adopted to enable fine interactive fusion of information between the image modality and the text modality at different levels. The embodiment of the present invention enhances the expressive power of cross-modal representation, improves the accuracy of cross-modal retrieval, and provides more accurate and powerful technical support for complex multimodal tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0060] Figure 1 is a schematic diagram of a cross-modal retrieval model provided by an embodiment of the present invention;

[0061] Figure 2 is a schematic diagram of the training process of the cross-modal retrieval model provided by an embodiment of the present invention;

[0062] Figure 3 is a schematic diagram of a cross-modal retrieval device provided by an embodiment of the present invention;

[0063] Figure 4 is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0065] In order to illustrate the technical solution of the present invention, a specific embodiment is provided below for illustration.

[0066] Currently, cross-modal retrieval methods include:

[0067] (1) Cross-modal retrieval based on traditional neural networks: Deep learning, especially convolutional neural networks (CNNs) and recurrent neural networks (RNNs), is used to analyze and process data of different modalities, map data of different modalities to a common low-dimensional vector space, and encode the semantic information of the data into vector representations. In the low-dimensional vector space, the semantic information of the data is encoded as a vector representation, so that semantically similar data points are closer in the vector space, while dissimilar ones are farther away.

[0068] (2) Cross-modal retrieval based on pre-trained models: Using large-scale data to pre-train specific models is aimed at enhancing the model's ability to understand and process different data types. By pre-training the model on a large scale, the model can learn rich semantic representations and patterns, thereby better understanding the characteristics and meanings of various data. Through the above method, a large amount of data can be fully utilized to improve the generalization ability and adaptability of the model, thereby achieving better performance in cross-modal retrieval tasks in specific fields.

[0069] However, in existing multimodal tasks, the modal representations of images and texts are often not effectively integrated. Although existing methods can capture the features of images and texts to a certain extent, they still have the following deficiencies when dealing with complex cross-modal tasks:

[0070] (1) Modal representations are difficult to align: In existing multimodal tasks, image and text modalities usually generate independent high-dimensional vector representations, and there is a lack of a unified mechanism to map them to the same spatial dimension. This misalignment problem makes it difficult for image and text modalities to fully capture the potential semantic associations between each other when they are fused, which in turn affects the overall performance of multimodal tasks. Since the representations are not synchronized or consistent between different modalities, the model exhibits low accuracy when processing cross-modal information and cannot effectively complete complex tasks such as image-text retrieval and description generation.

[0071] (2) Limitations of information interaction between modalities: Current multimodal fusion methods usually lack flexible and efficient mechanisms to achieve information interaction between image modalities and text modalities. Although they can capture the characteristics of each modality, it is often difficult to perform fine-grained and hierarchical interactions in the process of information transfer between modalities. Specifically, the representations of different modalities are usually processed in parallel, and the synergy between modalities is not fully utilized. This limitation makes it impossible for the model to achieve deep fusion of information when fusing multimodal information, which affects the performance of the model, especially when dealing with complex semantic relationships or cross-modal tasks.

[0072] (3) Missing matches: In actual multimodal datasets, the potential semantic associations between images and texts are often not clearly labeled, which makes cross-modal tasks prone to “missing matches” during training. When the model processes incompletely labeled data, images and texts that should be semantically matched may be mistakenly judged as unrelated. This problem can have a serious impact on the model’s learning process, causing the model to push related images and texts away from each other during training, making it unable to accurately reflect cross-modal semantic associations, and thus affecting the model’s generalization ability and overall performance. This problem is particularly prominent when multimodal data is scarce or incompletely labeled.

[0073] In this regard, the embodiment of the present invention proposes a cross-modal retrieval method, and constructs the following Figure 1 The cross-modal retrieval model shown in Figure 2 is used to achieve accurate cross-modal retrieval.

[0074] See also Figure 1 , the cross-modal retrieval method of this embodiment is described:

[0075] Step S101, encode the image modality through an image encoder, and encode the text modality through a text encoder to generate a vector representation in a high-dimensional space.

[0076] In this embodiment, an image encoder Visual Encoder and a text encoder Tex Encoder are constructed to encode the image modality and the text modality respectively, generating vector representations in a high-dimensional space. Subsequently, the vectors of these different modalities are passed through a shared self-attention layer Self Attention and a feed-forward neural network Feed Forward to capture the rich information inside the modality.

[0077] In terms of the choice of image encoder, the CLIP model can be used as the image encoder because the image encoder of the CLIP model has powerful capabilities in visual language tasks. For each input two-dimensional image I∈R H×W×C , where (H, W) is the resolution of the input image and C is the number of channels, CLIP reconstructs it into a flattened 2D patch sequence where (P, P) is the resolution of the image patch, is the number of patches. Similar to BERT, CLIP also adds a [class] tag before the patch sequence and uses a learnable 1D position embedding Where D v is the dimension of the image encoder. The input image representation can be calculated as follows:

[0078] V0=[E[class];p1W p ;...;p N W p ]+V pos

[0079] In terms of text encoder, RoBERTa can be used as the text encoder because it performs well in a wide range of natural language processing tasks. Each input sequence w is tokenized by byte-level byte pair encoding. [ <s> ]and[< / s> ] tag. The text representation of the input can be expressed as:

[0080]

[0081] in, E is the word embedding matrix, M is the number of tokens, and D t is the dimension of the text encoder, T pos is the position embedding matrix. Similarly, in this embodiment, the text encoder of the lth layer is represented as EncoderT l , the representation of the lth layer can be expressed as:

[0082] T l =EncoderT l (T l-1 ), l=1...L T

[0083] Among them, L T is the number of layers of the text encoder.

[0084] Step S102: input the vector of the image modality and the vector of the text modality into the same shared self-attention layer for alignment, and extract features through their respective feedforward network layers.

[0085] In order to make the vector representation of image and text in the same spatial dimension, this embodiment uses a shared self-attention layer to further align the representation of image modality and text modality in high-dimensional space. After the shared self-attention layer, the representation of image modality and text modality is updated as follows:

[0086] V′=SelfAttention(V)

[0087] T′=SelfAttention(T)

[0088] Then, in order to distinguish the representations of different modal vectors after the shared self-attention layer output, this embodiment uses different feed-forward network layers Feed Forward to update the text vector and image vector. These updated representations are further extracted through the feed-forward neural network:

[0089] V″=FeedForward(V′)

[0090] T″=FeedForward(T′)

[0091] Step S103, through the cross-attention mechanism, the vector of the image modality and the vector of the text modality are cross-modally fused to obtain a cross-modal representation vector of the image modality and a cross-modal representation vector of the text modality.

[0092] After being processed by their respective feedforward networks, this embodiment fuses the outputs of the two modalities through a cross-attention mechanism, so that the connection between the modalities can be fully interactive.

[0093] First, the information processed by the feedforward network is input into the self-attention module, where the output of each layer of the self-attention module is not only used for subsequent processing of the current modality, but also serves as the input of the cross-attention module of the other modality.

[0094]

[0095] Specifically, in cross-modal interaction, the self-attention output of the image modality will serve as the key and value of the cross-attention module in the text modality, while the self-attention output of the text modality Then, is used as the key and value of the cross-attention module of the image modality. Through this design, information from different modalities can be fused in a fine-grained manner at multiple levels, thereby enhancing the expressive power of cross-modal representation.

[0096] The cross-attention mechanism is implemented by the following formula:

[0097]

[0098] Finally, through the cross-attention mechanism, the cross-modal representation vector of the image modality is recorded as Z V , the cross-modal representation vector of the text modality is denoted as Z T These vectors not only contain deep information of their respective modalities, but also incorporate complementary information from other modalities, thereby improving the model's performance in multimodal tasks. This mechanism helps to more accurately capture the association between image modality and text modality, making cross-modal representation richer and more powerful, and better able to cope with complex multimodal challenges.

[0099] Step S104: output the similarity between the image modality and the text modality based on the cross-modal representation vector of the image modality and the cross-modal representation vector of the text modality.

[0100] The embodiment of the present invention encodes the image modality through an image encoder, encodes the text modality through a text encoder, and uses a shared self-attention mechanism to achieve alignment of high-dimensional vector representations between different modalities, which can fully extract and fuse deep information of different modalities. At the same time, the cross-attention mechanism is adopted to enable fine interactive fusion of information between the image modality and the text modality at different levels. The embodiment of the present invention enhances the expressive power of cross-modal representation, improves the accuracy of cross-modal retrieval, and provides more accurate and powerful technical support for complex multimodal tasks.

[0101] The above embodiment is a processing flow of establishing a cross-modal retrieval model. Before actual use, the cross-modal retrieval model needs to be pre-trained.

[0102] Considering that in actual operation, there may be potential semantic associations between unpaired images and texts in the same batch, but these associations are not labeled in the dataset. This situation is called the "cross-modal matching missing problem", which causes semantically matched images and texts to be mistakenly pushed apart during training. To solve this problem, this embodiment proposes a cross-modal soft label alignment method.

[0103] See also Figure 2 As shown in Figure 2, the training process of the cross-modal retrieval model includes:

[0104] Two single-modal pre-trained models are introduced as teacher models for image modality and text modality respectively;

[0105] Extracting image features from the image modalities of the training set according to the teacher model of the image modality;

[0106] Extracting text features from the text modality of the training set according to the teacher model of the text modality;

[0107] Based on the image features and text features, the similarity distribution soft labels of the image modality and text modality in the training set are calculated;

[0108] Distribute soft labels according to similarity and train a cross-modal retrieval model.

[0109] In this embodiment, two unimodal pre-trained models are introduced as teacher models to calculate soft labels to guide the training of the cross-modal retrieval model. This embodiment selects Unicom as the ImageTeacher Model of the image modality and selects Sentence-BERT as the Test Teacher Model of the text modality. Unicom is an advanced model for image retrieval, while Sentence-BERT is a model that performs well in semantic text similarity tasks. During the training process, this embodiment uses Unicom to extract image features from all image modalities in the dataset, and uses MPNet fine-tuned by Sentence-BERT to extract text features from all available text modalities. The extraction of image features and text features can be done offline, so there is no increase in complexity when training the cross-modal retrieval model online. It is worth noting that the selection of teacher models for image modality and text modality is flexible and can be replaced with any available model.

[0110] According to the features obtained by the teacher model, the cosine similarity between the image features and the text features is calculated, and the cosine similarity is normalized within the batch to obtain the similarity distribution P from the image modality to the text modality v2t and the similarity distribution P from text modality to image modality t2v.

[0111] Next, the cross-modal retrieval model is used to output the similarity distribution Q from the image modality to the text modality based on the training set. v2t and the similarity distribution Q from text modality to image modality t2v .

[0112] The similarity distribution from image modality to text modality can be expressed as:

[0113]

[0114] The similarity distribution from text modality to image modality can be expressed as:

[0115]

[0116] During training, the soft label distribution P of images and text is used v2t and P t2v To guide the corresponding learnable distribution Q v2t and Q t2v , and use KL divergence as the loss function:

[0117]

[0118] Among them, L CSA is the KL divergence, D KL is the preset coefficient.

[0119] This improved method ensures that during the training process, not only the deep representation of unimodal information is taken into account, but also the effective sharing and alignment of cross-modal information is guided through the soft label mechanism, thereby improving the performance of the model in multimodal tasks.

[0120] In combination with the above content, this embodiment solves the following advantages:

[0121] (1) Cross-modal representation alignment problem: In order to solve the problem of inconsistent representation of image and text modalities in high-dimensional space, this embodiment proposes a mechanism of sharing self-attention layer. Through this mechanism, the high-dimensional vector representation of image modality and text modality can be more accurately aligned in the same space, making information sharing and complementarity between different modalities smoother and more effective. By sharing the self-attention layer, the features of image and text can not only capture the rich information within each modality, but also be synchronously represented in high-dimensional space, thereby improving the quality of information fusion in cross-modal tasks. This representation alignment not only improves the model's ability to capture semantic associations between modalities, but also enhances its performance in complex tasks such as cross-modal retrieval and image-text generation. Through the shared self-attention mechanism, the high-dimensional space representations of image and text modalities can be effectively aligned, ensuring that the information between modalities maintains the feature differences of each modality on the basis of deep fusion, thereby improving the accuracy and consistency of cross-modal representation.

[0122] (2) Problem of deep interaction of modal information: In order to further enhance the information interaction between image and text modalities, this embodiment introduces a cross-attention mechanism. This mechanism enables image and text modalities to share and transfer information in multi-level fine-grained interactions, thereby achieving more comprehensive information fusion between modalities. In each self-attention layer, the representation of different modalities is not only used for its own update processing, but also as information input for another modality, and deep interaction between modalities is achieved through the cross-attention module. This design greatly enhances the richness and comprehensiveness of information transfer in cross-modal tasks, ensuring that the model can more effectively utilize information from different modalities, thereby improving overall task performance.

[0123] (3) Missing cross-modal matching problem: In cross-modal datasets, unlabeled semantic associations often lead to missing matching problems between image and text modalities during training. In order to meet this challenge, this embodiment proposes a cross-modal soft label alignment mechanism. By introducing pre-trained unimodal teacher models, such as Unicom and Sentence-BERT, soft labels for images and texts are generated to make up for the potential semantic association problem between unpaired images and texts, and guide the cross-modal retrieval model to better align unlabeled data during training. This mechanism allows the model to not only train based on unimodal information when processing cross-modal tasks, but also guide the model to mine potential semantic associations between modalities with the help of soft labels, avoiding erroneously pushing away images and texts that should be matched, thereby improving the generalization ability and overall accuracy of the model. This method ensures that the model has stronger adaptability and robustness when processing large-scale cross-modal data, avoids the loss of semantic matching information during training, and improves the accuracy of cross-modal information alignment.

[0124] (4) The entire scheme of this embodiment enhances the expressive power of cross-modal representation through the synergy of shared self-attention, cross-attention and soft label alignment, providing more accurate and powerful technical support for complex multimodal tasks.

[0125] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0126] Figure 3 is a schematic diagram of the structure of a cross-modal retrieval device 30 provided in an embodiment of the present invention, see Figure 3 As shown, the device comprises:

[0127] The encoding module 31 is used to encode the image modality through an image encoder and encode the text modality through a text encoder to generate a vector representation in a high-dimensional space.

[0128] The alignment module 32 is used to input the vector of the image modality and the vector of the text modality into the same shared self-attention layer for alignment, and extract features through their respective feedforward network layers.

[0129] The cross module 33 is used to perform cross-modal cross-fusion processing on the vector of the image modality and the vector of the text modality through a cross-attention mechanism to obtain a cross-modal representation vector of the image modality and a cross-modal representation vector of the text modality.

[0130] The output module 34 is used to output the similarity between the image modality and the text modality based on the cross-modal representation vector of the image modality and the cross-modal representation vector of the text modality.

[0131] In a possible implementation, the cross-connect module 33 is used to:

[0132] The vector of the image modality is input into the image self-attention module, and the vector of the text modality is input into the text self-attention module;

[0133] The outputs of the image self-attention module and the text self-attention module are input into the image cross-attention module to obtain the cross-modal representation vector of the image modality;

[0134] The outputs of the image self-attention module and the text self-attention module are input into the text cross-attention module to obtain the cross-modal representation vector of the text modality.

[0135] In a possible implementation, the cross-modal retrieval device further includes: a training module;

[0136] The training modules are used to:

[0137] Two single-modal pre-trained models are introduced as teacher models for image modality and text modality respectively;

[0138] Extracting image features from the image modalities of the training set according to the teacher model of the image modality;

[0139] Extracting text features from the text modality of the training set according to the teacher model of the text modality;

[0140] Based on the image features and text features, the similarity distribution soft labels of the image modality and text modality in the training set are calculated;

[0141] Distribute soft labels according to similarity and train a cross-modal retrieval model.

[0142] In one possible implementation, the training module is used to:

[0143] Calculate the loss function based on the similarity distribution of soft labels and the similarity distribution output by the cross-modal retrieval model based on the training set;

[0144] According to the loss function, adjust the model parameters of the cross-modal retrieval model;

[0145] Continue to iterate until the preset iteration termination condition is met.

[0146] As a possible implementation, the loss function is the KL divergence function:

[0147]

[0148] Among them, L CSA is the KL divergence, D KL is the preset coefficient, P v2t and P t2v They are the similarity distribution from image modality to text modality and the similarity distribution from text modality to image modality in the similarity distribution soft label, respectively. v2t and Q t2v They are respectively the similarity distribution from image modality to text modality and the similarity distribution from text modality to image modality output by the cross-modal retrieval model based on the training set.

[0149] As a possible implementation, the image encoder is a CLIP model and the text encoder is a RoBERTa model.

[0150] As a possible implementation, the teacher model for the image modality is the Unicom model, and the teacher model for the text modality is the Sentence-BERT model.

[0151] The embodiment of the present invention encodes the image modality through an image encoder, encodes the text modality through a text encoder, and uses a shared self-attention mechanism to achieve alignment of high-dimensional vector representations between different modalities, which can fully extract and fuse deep information of different modalities. At the same time, the cross-attention mechanism is adopted to enable fine interactive fusion of information between the image modality and the text modality at different levels. The embodiment of the present invention enhances the expressive power of cross-modal representation, improves the accuracy of cross-modal retrieval, and provides more accurate and powerful technical support for complex multimodal tasks.

[0152] Figure 4 FIG. 4 is a schematic diagram of an electronic device 40 provided by an embodiment of the present invention. Figure 4 As shown, the electronic device 40 of this embodiment includes: a processor 41, a memory 42, and a computer program 43 stored in the memory 42 and executable on the processor 41, such as a cross-modal retrieval program. When the processor 41 executes the computer program 43, the steps in the above-mentioned cross-modal retrieval method embodiments are implemented. Alternatively, when the processor 41 executes the computer program 43, the functions of the modules in the above-mentioned device embodiments are implemented, such as Figure 3 The functions of the modules 31 to 34 are shown.

[0153] Exemplarily, the computer program 43 may be divided into one or more modules / units, which are stored in the memory 42 and executed by the processor 41 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program 43 in the electronic device 40.

[0154] The electronic device 40 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device 40 may include, but is not limited to, a processor 41 and a memory 42. Those skilled in the art will appreciate that Figure 4 It is only an example of the electronic device 40 and does not constitute a limitation of the electronic device 40. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 40 may also include input and output devices, network access devices, buses, etc.

[0155] The processor 41 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0156] The memory 42 may be an internal storage unit of the electronic device 40, such as a hard disk or memory of the electronic device 40. The memory 42 may also be an external storage device of the electronic device 40, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 40. Further, the memory 42 may also include both an internal storage unit of the electronic device 40 and an external storage device. The memory 42 is used to store the computer program and other programs and data required by the electronic device 40. The memory 42 may also be used to temporarily store data that has been output or is to be output.

[0157] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0158] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0159] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0160] In the embodiments provided by the present invention, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0161] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0162] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0163] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0164] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A cross-modal retrieval method, characterized in that: include: Encode the image modality through the image encoder and encode the text modality through the text encoder to generate a vector representation in a high-dimensional space; Inputting the vector of the image modality and the vector of the text modality into the same shared self-attention layer for alignment, and extracting features through their respective feed-forward network layers; Through a cross-attention mechanism, a cross-modal cross-fusion process is performed on the vector of the image modality and the vector of the text modality to obtain a cross-modal representation vector of the image modality and a cross-modal representation vector of the text modality; Based on the cross-modal representation vector of the image modality and the cross-modal representation vector of the text modality, the similarity between the image modality and the text modality is output.

2. The cross-modal retrieval method according to claim 1, characterized in that: The cross-attention mechanism is used to perform cross-modal cross-fusion processing on the vector of the image modality and the vector of the text modality to obtain the cross-modal representation vector of the image modality and the cross-modal representation vector of the text modality, including: Inputting the vector of the image modality into an image self-attention module, and inputting the vector of the text modality into a text self-attention module; Inputting the outputs of the image self-attention module and the text self-attention module into the image cross-attention module to obtain a cross-modal representation vector of the image modality; The outputs of the image self-attention module and the text self-attention module are input into the text cross-attention module to obtain a cross-modal representation vector of the text modality.

3. The cross-modal retrieval method according to claim 1 or 2, characterized in that: Also includes: The training process of the cross-modal retrieval model; The training process of the cross-modal retrieval model includes: Two single-modal pre-trained models are introduced as teacher models for image modality and text modality respectively; Extracting image features from the image modalities of the training set according to the teacher model of the image modality; Extracting text features from the text modality of the training set according to the teacher model of the text modality; Calculate similarity distribution soft labels of image modality and text modality in a training set according to the image features and the text features; The cross-modal retrieval model is trained according to the similarity distribution soft labels.

4. The cross-modal retrieval method according to claim 3, characterized in that: The step of training the cross-modal retrieval model according to the similarity distribution soft labels includes: Calculating a loss function according to the similarity distribution soft labels and the similarity distribution output by the cross-modal retrieval model according to the training set; According to the loss function, adjusting model parameters of the cross-modal retrieval model; Continue to iterate until the preset iteration termination condition is met.

5. The cross-modal retrieval method according to claim 4, characterized in that: The loss function is the KL divergence function: Among them, L CSA is the KL divergence, D KL is the preset coefficient, P v2t and P t2v are respectively the similarity distribution from image modality to text modality and the similarity distribution from text modality to image modality in the similarity distribution soft label, Q v2t and Q t2v They are respectively the similarity distribution from image modality to text modality and the similarity distribution from text modality to image modality output by the cross-modal retrieval model according to the training set.

6. The cross-modal retrieval method according to claim 1, characterized in that: The image encoder is a CLIP model, and the text encoder is a RoBERTa model.

7. The cross-modal retrieval method according to claim 3, characterized in that: The teacher model for the image modality is the Unicom model, and the teacher model for the text modality is the Sentence-BERT model.

8. A cross-modal retrieval device, characterized in that: include: An encoding module, used for encoding the image modality through an image encoder and encoding the text modality through a text encoder to generate a vector representation in a high-dimensional space; An alignment module, used for inputting the vector of the image modality and the vector of the text modality into the same shared self-attention layer for alignment, and extracting features through their respective feedforward network layers; A cross module, used to perform cross-modal cross-fusion processing on the vector of the image modality and the vector of the text modality through a cross-attention mechanism to obtain a cross-modal representation vector of the image modality and a cross-modal representation vector of the text modality; An output module is used to output the similarity between the image modality and the text modality based on the cross-modal representation vector of the image modality and the cross-modal representation vector of the text modality.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images