An image-text cross-modal retrieval model, method and computer device based on local generation double driving

Abstract

The application discloses a kind of based on local generation double drive's image text cross-modal retrieval model, method and computer equipment, comprising: S1, the pair of original image and text in training data is respectively extracted using encoder feature, obtains the global feature and local feature of original image and text;S2, input image and text global feature, local feature, through local drive semantic completion module screening obtains the local detail feature that global feature ignores, then local detail feature and global feature are fused, obtain the fusion feature after semantic completion;S3, input text, through generation drive semantic conversion module obtains the generation image feature with text semantic information;S4, the global feature of image and text global feature obtained by S1, the image fusion feature and text fusion feature obtained by S2, the generation image feature and image global feature obtained by S3 and text global feature between similarity are calculated and training is carried out using contrast loss function.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and natural language processing, specifically to a cross-modal image and text retrieval model, method, and computer device based on local generation dual-driven approach. Background Technology

[0002] Images and text are two key information carriers that help humans and intelligent applications better understand the real world. Extensive research in computer vision and natural language processing has been conducted to learn and understand these two modalities. Image-text cross-modal retrieval, as a fundamental yet complex research direction, aims to uncover relationships between samples from different modalities—that is, to retrieve samples from one modality with similar semantics using samples from another. The development of this technology is of great significance for various practical applications, such as people search, sketch-based image retrieval, and food recipe retrieval.

[0003] Existing image-text cross-modal retrieval methods can be divided into two categories: score-based matching methods and embedding-based matching methods. Score-based matching methods focus on the cross-modal interaction between local visual features and text features, and then calculate a cumulative similarity score. Embedding-based matching methods employ a dual-encoder architecture, using two encoders to extract global features from the image and text respectively, and then learning an image-text joint embedding space by constraining the alignment between the global features of the image and the text. However, while score-based matching methods have an advantage in accuracy due to the large depth of interaction between modalities, their computational efficiency and response speed are often unsatisfactory when dealing with massive amounts of data. In contrast, embedding-based matching methods use dual encoders to process images and text, eliminating direct interaction between modalities and significantly improving retrieval efficiency. However, this coarse-grained alignment limits retrieval performance.

[0004] Existing methods focus too much on cross-modal interaction and coarse-grained alignment between modalities, neglecting the ability to capture fine-grained semantic information without reducing retrieval efficiency and mitigating the significant differences in the expression and understanding of the same concept caused by the heterogeneity of the underlying structure between image and text modalities. Therefore, to address these two challenges, this invention designs an image-text cross-modal retrieval model based on local generation dual-drive to improve the model's ability to capture fine-grained semantic information and mitigate modal differences. Summary of the Invention

[0005] To address the limitations of existing methods in capturing fine-grained semantic information and mitigating modal discrepancies, this invention proposes a local generation-driven, dual-modal image-text retrieval model. This model comprises two main modules: a local-driven semantic completion module and a generation-driven semantic transformation module. The local-driven semantic completion module filters and supplements neglected local fine-grained information for global features, resulting in a feature representation incorporating this information. Simultaneously, the generation-driven semantic transformation module generates image features with textual semantic information. These features are then used to narrow the relative positions of image and text within the image-text joint embedding space, thereby reducing modal discrepancies.

[0006] A cross-modal image-text retrieval model based on local generation dual-drive is obtained through the following steps:

[0007] S1, use the encoder to extract features from the pairs of original images and text in the training data to obtain the global features, local features, global features, and local features of the original images and text;

[0008] S2, input image and text global features and local features, through the local driven semantic completion module to filter out local detail features ignored by the global features, and then fuse the local detail features with the global features to obtain the semantically completed fused features;

[0009] S3, Input text, through the generation-driven semantic transformation module, obtain generated image features with text semantic information;

[0010] S4 calculates the similarity between the global image features and global text features obtained in S1, the image fusion features and text fusion features obtained in S2, the generated image features and global image features obtained in S3, and the global text features, and then uses the contrastive loss function for training.

[0011] As a preferred technical solution, the feature extraction process in step S1 includes:

[0012] S1.1: Given an image V, first slice it into multiple blocks to form a block sequence, and then pass it through a trainable linear projection function, denoted as V={v1,v2,,,,,v... n}, where n represents the number of blocks. Then, class labels [CLS] are inserted into the sequence to learn the global representation. Position embeddings are then added to each block to encode spatial information. Next, the sequence is input into an image encoder to obtain image features. The image encoder is composed of L... v Composed of a visual transformer layer, making Let l be the output sequence of the l-th visual transformer layer, and serve as the input sequence for the l+1 layer. The output sequence of the l+1 layer is then calculated using the following formula:

[0013]

[0014] in, Let L represent the image features output by the l-th layer, MHSA(·) denotes a multi-head self-attention layer, MLP(·) denotes a multilayer perceptron network, and LN(·) denotes layer normalization. This represents the image features after passing through the l-th multi-head self-attention layer and layer normalization, followed by residual connections. Finally, the image encoder outputs the final image features. in f represents the global features of the image. i v Let G represent the i-th local feature of the image. Projecting the global image features onto the image-text joint embedding space yields the final global image features, G. v .

[0015] S1.2: Given text T, convert the text into a sequence of tags, represented as T = {t1, t2, ..., t}. m}, where m represents the number of marker sequences. Then, the text is enclosed in [SOS] and [EOS] markers to indicate the beginning and end of the text. The marker sequence T = {t} sos ,t1,t2,…,t m ,t eos The input is fed into the text encoder Transformer, which learns the correlations between words through masked self-attention to obtain the extracted text features. Ultimately, the features marked with [EOS] will be... Projecting the image into the image-text joint embedding space yields the final global text feature G. t .

[0016] As a preferred technical solution, step S2 involves using a local-driven semantic completion module to filter out local detail features ignored by the global features, and then fusing the local detail features with the global features to obtain the semantically completed fused features.

[0017] S2.1: The local-driven semantic completion module proposed in this invention consists of two parts: an explicit filtering method and an implicit filtering method. The global and local features of the image and text extracted in S1 are input into the explicit and implicit filtering methods to obtain two sets of local detail features ignored by the global features of the image and text. Then, the local detail features are fused with the global features to obtain the fused feature representation.

[0018] S2.2: Using the explicit filtering method, first calculate the similarity between the global and local features of the image obtained in S1, and then obtain... Then, sort S in ascending order. v ∈R 1×n The local features with the lowest similarity are sorted and selected. These K features are local details ignored by the global image features obtained through explicit filtering. Then, the local details are fused with the global image features to obtain the explicit image fusion features with semantic completion through explicit filtering. Where d represents the dimension of the global and local features of the image, and 2d represents the dimension of the fused features. The calculation formula is as follows:

[0019]

[0020] Among them, the feature sort n (·) indicates a sorting operation. Represents the Kth similarity score in the sorted similarity matrix, Concat d (·) indicates a feature concatenation operation, Mean k (·) indicates the average pooling operation.

[0021] S2.3: Using an implicit filtering method, select the local image features obtained in S1. Sort the features in descending order along dimension d, and select the top M local detail features with the largest responses. These M features are local detail features that were ignored by the global image features obtained through implicit filtering. Then, the local detail features and the global image features are fused to obtain implicit image fusion features with semantic completion through implicit filtering. The calculation formula is:

[0022]

[0023] in, This represents local detail features obtained through implicit filtering. It is the local features of the image sorted in descending order along the direction of dimension d.

[0024] S2.4: Text Applying the same process, through both explicit and implicit filtering methods, we obtain the displayed text fusion features. Features of fusion with implicit text

[0025] As a preferred technical solution, the process of obtaining generated image features with textual semantic information through the generation-driven semantic transformation module in step S3 includes:

[0026] S3.1: Given text T, input it into a pre-trained text-image generation model f θ In this process, a generated image feature T containing textual semantic information is obtained. gen The calculation formula is:

[0027] withe~[1,E]

[0028] Where e represents the iteration time, and a total of E iterations are required. This represents the image embedding generated at time e. This represents Gaussian noise from random sampling.

[0029] S3.2: Generate image features T gen Input a projection layer MLP s (·), T gen Projected into the image-text joint embedding space, MLP s The structure and parameters of (·) are shared with the last projection layer used in the image encoder. The calculation formula is:

[0030]

[0031] in This represents the generated image features with textual semantic information projected into the image-text joint embedding space. To ensure retrieval efficiency, the generation-driven semantic transformation module is only used during training and not during application.

[0032] As a preferred technical solution, step S4 specifically includes:

[0033] S4.1: Global image features G obtained from S1 v and text global features G t Calculate similarity, then apply constraints using a loss function, which is:

[0034]

[0035] in

[0036]

[0037] in, Represents the global features of the i-th image. This represents the i-th global feature of the text. This represents the transpose operation, N represents the number of matched image-text pairs, and τ1 represents the temperature parameter, which is a trainable variable. This represents the original contrast loss, mainly composed of and It consists of two sub-items. and This represents the contrast loss between image-to-text and text-to-image.

[0038] S4.2: At the same time, the displayed image fusion features obtained in S2 through explicit and implicit filtering methods are... and display text fusion features Implicit image fusion features Features of fusion with implicit text The same loss function is used between them. Specifically, Global features G in the image v and text global features G t Replace with display image fusion features and display text fusion features Obtain the display local driving contrast loss Then Global features G in the image v and text global features G t Replace with implicit image fusion features Features of fusion with implicit text Obtain implicit local driving contrast loss

[0039] S4.3: Similarly, for the generated image features obtained in S3 and global features of the image G v and text global features G t Constraints are applied using a contrastive loss function, which is:

[0040]

[0041] in

[0042]

[0043] in, Let represent the i-th generated image feature, and τ2 represent the trainable temperature hyperparameter. This represents the generation-driven contrastive loss, mainly composed of and It consists of two sub-items. and This represents the contrast loss between generated image to text and generated image to image.

[0044] Finally, during training, a total loss function is designed using the original contrastive loss, explicit local-driven contrastive loss, implicit local-driven contrastive loss, and generative-driven contrastive loss, and unified joint training is performed. The calculation formula is as follows:

[0045]

[0046] Here, α, β, and γ are hyperparameters that balance the four loss terms.

[0047] Based on the above-mentioned image-text cross-modal retrieval model, this invention also proposes an image-text cross-modal retrieval method based on local generation dual-drive. The image-text cross-modal retrieval method is implemented using the above-mentioned image-text cross-modal retrieval model, and the specific process is as follows:

[0048] Given images and text in a dataset, input image encoders and text encoders extract global and local features of the images and text. Then, through explicit and implicit filtering methods of the local-driven semantic completion module, local detail features ignored by the global features are filtered and fused, resulting in explicit and implicit fused features of the images and text, respectively. Finally, cosine similarity is calculated using the global features of the images and text, the explicit image fusion features and explicit text fusion features, and the implicit image fusion features and implicit text fusion features, and the retrieval results are returned. For example, the cosine similarity calculation of global features of the images and text is as follows:

[0049]

[0050] Where n1 and n2 represent the number of images and text. The cosine similarity scores of the displayed image fusion features, displayed text fusion features, implicit image fusion features, and implicit text fusion features can be calculated by replacing the global image features and global text features in the formula with the corresponding features.

[0051] Furthermore, both the aforementioned cross-modal image and text retrieval models and methods can be stored in computer devices, executed as instructions, or stored as program code.

[0052] The beneficial effects of this invention are:

[0053] (1) A local-driven semantic completion module is proposed. Through explicit and implicit filtering methods, it can capture local details ignored by global features more comprehensively, and then integrate the ignored local details with global features to supplement semantic information.

[0054] (2) A generative-driven semantic transformation module is proposed, which uses a generative model to obtain generated image features with textual semantic information. These features serve as a bridge between modalities, bringing the relative positions of image and text in the image-text joint embedding space closer, thereby reducing modal differences. This module is only used during training, avoiding additional time waste and ensuring retrieval efficiency.

[0055] (3) Compared with existing methods, this method can effectively improve the model's ability to capture local detail information and reduce modal differences, thereby improving the accuracy of cross-modal image and text retrieval. Applying this invention to the CLIP_Vit-B / 16 and CLIP_Vit-L / 14 retrieval models, it improved the RSUM index by 10.7% and 7.5% respectively on the Flickr30k dataset. On the MS-COCO dataset, it improved the RSUM index by 10.8% and 5.7% respectively. Attached Figure Description

[0056] Figure 1 This is a framework diagram of an image-text cross-modal retrieval model based on local generation dual-drive; Detailed Implementation

[0057] This invention first extracts features from the original image and text, obtaining global and local features respectively. Then, using these global and local features as input, and employing explicit and implicit filtering methods in a locally driven semantic completion module, explicit and implicit fusion features of the image and text are obtained, respectively. Next, a text-image generation model is used to generate image features with textual semantic information based on the input text. Finally, a unified joint training process is performed, applying the original contrast loss to the global image and text features, the explicit local driven contrast loss to the explicit image and text fusion features, the implicit local driven contrast loss to the implicit image and text fusion features, and the generation-driven contrast loss to the generated image features and the global image and text features.

[0058] The invention will now be further described with reference to the accompanying drawings.

[0059] Figure 1 This is a framework diagram of the locally generated dual-driven image-text cross-modal retrieval model proposed in this invention. First, feature extraction is performed on the original image and text to obtain global and local features, respectively. Then, using the global and local features as input, the explicit and implicit filtering methods of the locally driven semantic completion module are used to obtain explicit and implicit fusion features of the image and text, respectively. Next, a text-image generation model is used to generate image features with textual semantic information based on the input text. Finally, the original contrast loss is applied to the global image features and global text features; the explicit local-driven contrast loss is applied to the explicit image fusion features and explicit text fusion features; the implicit local-driven contrast loss is applied to the implicit image fusion features and implicit text fusion features; and the generation-driven contrast loss is applied to the generated image features and the global image features and global text features. Finally, a unified joint training is performed. The specific process is as follows:

[0060] S1, Feature Extraction: Extracting features from the original image and text;

[0061] Specific implementation: Given an image V and text T, input them into the image encoder and text encoder to obtain the global features G. v G t and local features In the implementation example, image features can be extracted using the CLIP image encoder pre-trained on a large corpus to obtain global and local image features, calculated as follows:

[0062]

[0063] in, The output sequence of the l-th visual transformer layer is used as the input sequence of the l+1-th layer. The output sequence of the layer, MHSA(·) represents a multi-head self-attention layer, MLP(·) represents a multilayer perceptron, and LN(·) represents layer normalization.

[0064] Text features were also extracted using CLIP text encoders pre-trained on a large corpus to obtain global and local text features. All input images were resized to 224*224 pixels, and the length of the text descriptions was uniformly set to 77. Each batch contained 32 image-text pairs.

[0065] S2, semantic information completion: such as Figure 1 As shown, by using two filtering methods of the local-driven semantic completion module, two sets of local detail features that are ignored by the global features are obtained. Then, the local detail features are fused with the global features to obtain two sets of fused features after semantic completion.

[0066] Specific implementation: First, using a visible filtering method, similarity calculation is performed on the global and local features of the image obtained from S1 to obtain... Then, sort S in ascending order. v ∈R 1×n The local features with the lowest similarity are sorted and selected. These K features are local details ignored by the global image features obtained through explicit filtering. Then, the local details are fused with the global image features to obtain the explicit image fusion features with semantic completion through explicit filtering. The calculation formula is:

[0067]

[0068] Among them, sort n (·) represents a sorting operation, Concat d(·) indicates a feature concatenation operation, Mean k (·) indicates the average pooling operation. (Text) Applying the same display filtering process, we obtain the display text fusion features. In the implementation example, K is set to 20. Twenty local features that are not similar to the global features are selected, average pooling is performed, and then they are fused with the global features to supplement semantic information.

[0069] Then, using an implicit filtering method, the local features of the image obtained by S1 are... Sort the features in descending order along dimension d, and select the top M local features with the largest responses. These M features are local detail features that are ignored by the global image features obtained through implicit filtering. Then, fuse the local detail features and the global image features to obtain implicit image fusion features with semantic completion through implicit filtering. The calculation formula is:

[0070]

[0071] in, This represents local detail features obtained through implicit filtering. (Text) Applying the same implicit filtering process, explicit and implicit text fusion features are obtained. In the implementation example, M is set to 5, and the five local detail features with the largest response values ​​are selected and then fused with the global features to supplement semantic information.

[0072] S3, generate image features with textual semantic information: such as Figure 1 As shown, based on the given text, the generated image features with textual semantic information are obtained through the generation-driven semantic transformation module;

[0073] Implementation details: Given text T, input it into a pre-trained text-image generation model f. θ In this process, a generated image feature with textual semantic information is obtained, calculated using the following formula:

[0074] withe~[1,E]

[0075] Where e represents the iteration time, This represents the image embedding generated at time e. This represents Gaussian noise from random sampling. Then, T... gen Input a projection layer MLP s (·), T gen Projected into the image-text joint embedding space, MLP sThe structure and parameters of (·) are shared with the last projection layer used in the image encoder. The calculation formula is:

[0076]

[0077] in This represents the generated image features with textual semantic information projected into the image-text joint embedding space. To ensure retrieval efficiency, the generation-driven semantic transformation module is only used during training and not during application. In this embodiment, the text-image generation model f... θ The DALL-E 2 text-image generation model was adopted.

[0078] S4, using contrastive loss constraints, performs joint training: (e.g.) Figure 1 As shown, the corresponding features obtained in steps S1, S2 and S3 are subjected to constrained training.

[0079] Specific implementation: The global image features G obtained from S1... v and text global features G t Calculate similarity, then apply a contrastive loss function as a constraint. The loss function is:

[0080]

[0081] in

[0082]

[0083] Where N represents the number of matched image-text pairs, and τ1 represents the temperature parameter, which is a trainable variable. This indicates the original contrast loss.

[0084] Then, the displayed image fusion features obtained by S2 through explicit and implicit filtering methods are analyzed. and display text fusion features Implicit image fusion features Features of fusion with implicit text Using the same loss function, the contrast loss of local driving is calculated. Loss compared to implicit local driving

[0085] Then, the generated image features obtained from S3 are analyzed. Global features of the graph G v and text global features G t Constraints are applied using a contrastive loss function, which is:

[0086]

[0087] in

[0088]

[0089] Where τ2 represents the trainable temperature hyperparameter, This represents the generation-driven contrastive loss. Finally, during training, the original contrastive loss, explicit local-driven contrastive loss, implicit local-driven contrastive loss, and generation-driven contrastive loss are used for unified joint training, calculated as follows:

[0090]

[0091] Here, α, β, and γ are hyperparameters used to balance the four loss terms. In the implementation example, α, β, and γ are set to 1, 0.98, and 0.01, respectively, to balance the loss terms.

[0092] When using the above model for cross-modal image and text retrieval, follow these steps:

[0093] Given images and text in a dataset, input image encoders and text encoders extract global and local features of the images and text. Then, through the explicit and implicit filtering methods of the local-driven semantic completion module, local detail features ignored by the global features are filtered and fused to obtain explicit and implicit fused features of the images and text, respectively. Finally, cosine similarity is calculated using global image features and global text features, explicit image fused features and explicit text fused features, and implicit image fused features and implicit text fused features, and the retrieval results are returned.

[0094] The aforementioned cross-modal image and text retrieval models and methods can be stored in computer devices, executed as instructions, or stored as program code.

[0095] The detailed descriptions listed above are merely specific descriptions of feasible embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. All equivalent methods or modifications that do not depart from the technology of the present invention should be included within the scope of protection of the present invention.

Claims

1. A cross-modal image-text retrieval method based on local generation dual-drive, characterized in that, Obtained by the following method: S1, use the image encoder and text encoder to extract features from the paired original images and text respectively, to obtain the global image features, local image features, global text features, and local text features of the original images and text; S2, input the global and local features of the image and text, obtain the local detail features ignored by the global features through local-driven semantic completion filtering, and then fuse the local detail features with the global features to obtain the semantically completed image fusion features and text fusion features; In step S2, the method for obtaining image fusion features includes the following: S2.1: Using the explicit filtering method, first calculate the similarity between the global and local features of the image obtained in S1, and then obtain... Then, sort by ascending order. The local features with the lowest similarity are sorted and selected. These K features are local details ignored by the global image features obtained through explicit filtering. Then, the local details are fused with the global image features to obtain the explicit image fusion features with semantic completion through explicit filtering. The calculation formula is: in, This indicates a sorting operation. This indicates a feature concatenation operation. This represents the average pooling operation; where The dimensions representing global and local features of an image. The dimension representing the fused features. This represents the Kth similarity score in the sorted similarity matrix; S2.2: Using an implicit filtering method, select the local image features obtained in S1. The features are sorted in descending order along dimension d, and the top M local detail features with the largest responses are selected. These M features are local detail features that are ignored by the global image features obtained through implicit filtering. Then, the local detail features and the global image features are fused to obtain implicit image fusion features with semantic completion through implicit filtering. The calculation formula is: in, This represents local detail features obtained through implicit filtering. These are the local features of the image sorted in descending order along the direction of dimension d; In step S2, the method for obtaining text fusion features is the same as the method for obtaining image fusion features, resulting in displayed text fusion features. Features of fusion with implicit text ; S3, during the training phase, input text and generate image features with textual semantic information through generation-driven semantic transformation; S3 is as follows: S3.1: Given text The input is fed into a pre-trained text-image generation model. In this process, a generated image feature with textual semantic information is obtained, calculated using the following formula: in, Indicates the iteration time. Indicates the first Image embedding generated at any time, This represents Gaussian noise from random sampling; S3.2: Will Input a multi-layer perceptron projection layer ,Will Projected into the image-text joint embedding space, the calculation formula is: in Represents generated image features with textual semantic information projected into the image-text joint embedding space; S4 calculates the similarity between the global image features and global text features obtained in S1, the image fusion features and text fusion features obtained in S2, the generated image features and global image features obtained in S3, and the global text features. Constraints are applied through their respective loss functions, and finally, the total loss function is used for training.

2. The image-text cross-modal retrieval method based on local generation dual-drive according to claim 1, characterized in that, The S1 feature extraction process includes: S1.1: Given an image as First, it is divided into multiple blocks, forming a block sequence, and then processed by a trainable linear projection function, denoted as: Here, n represents the number of blocks; then, class labels [CLS] are inserted into the sequence to learn the global representation; positional embeddings are then added to each block to encode spatial information; next, the sequence is input into an image encoder to obtain image features, where the image encoder is composed of... Composed of a visual transformer layer, making for The output sequence of the layered visual transformer layer, and used as The input sequence of the layer, then The output sequence of the layer is calculated using the following formula: in, This indicates the multi-head self-attention layer. Represents a multilayer sensing network. Presentation layer normalization; Finally, the image encoder outputs the image features. ,in, Represents global features of the image. Indicates the first The image local features are projected onto the image-text joint embedding space to obtain the final image global features. ; S1.2: The given text is First, the text is converted into a sequence of tokens, represented as... Then enclose the text in [SOS] and [EOS] tags to indicate the beginning and end of the text; then add the tag sequence. The input is fed into the Transformer text encoder, where word correlations are learned through masked self-attention to obtain the extracted text features. Ultimately, the features marked with [EOS] will be... Projecting these features into the image-text joint embedding space yields the final global text features. .

3. The image-text cross-modal retrieval method based on local generation dual-drive according to claim 1, characterized in that, The implementation of S4 includes: S4.1: Global features of the image obtained from S1 and global text features Calculate similarity, then apply a contrastive loss function as a constraint. The loss function is: in in, This indicates the number of matched image-text pairs. The temperature parameter is a trainable variable. Indicates the original contrast loss; S4.2: Calculate the displayed image fusion features obtained by S2 through explicit and implicit filtering methods. and display text fusion features Implicit image fusion features Features of fusion with implicit text The similarity between them is determined using the same loss function as in S4.1, and is expressed as the explicit local-driven contrast loss. Loss compared to implicit local driving ; S4.3: Features of the generated image obtained from S3 and global features of the image and global text features Constraints are applied using a contrastive loss function, which is: in in, This represents the trainable temperature hyperparameter. This indicates the generation of the driver comparison loss.

4. The image-text cross-modal retrieval method based on local generation dual-drive according to claim 3, characterized in that, The formula for calculating the total loss function of S4 is as follows: in, , and It is a hyperparameter that balances the four loss terms.

5. A cross-modal image-text retrieval method based on local generation dual-drive, characterized in that, The image-text cross-modal retrieval method described herein is implemented using the image-text cross-modal retrieval method of any one of claims 1-4, and the specific process is as follows: Given an image and text, first input the image encoder and text encoder respectively to extract global features of the image, global features of the text, local features of the image, and local features of the text; Then, by using the explicit and implicit filtering methods of local-driven semantic completion, local detail features ignored by global features are filtered and fused to obtain the explicit fusion features and implicit fusion features of images and text respectively. Finally, cosine similarity is calculated using global image features and global text features, explicit image fusion features and explicit text fusion features, implicit image fusion features and implicit text fusion features, and the retrieval results are returned.

6. A computer device, characterized in that, It stores the program of any one of claims 1-4 and is capable of executing the program of the image-text cross-modal retrieval method of claim 5.

Images