A method and system for image-text retrieval based on coarse and fine-grained modal interaction

Abstract

The application discloses a kind of based on thick and thin granularity modal interaction's picture-text retrieval method and system, comprising: extracting image global feature to sample image, extracting the word fine-grained feature of each word and text context feature to sample text;According to the similarity between image global feature and text context feature, obtain global cross-modal similarity;According to image global feature and the word fine-grained feature of each word, obtain local cross-modal similarity;Design multi-scale modal contrast loss function, according to global cross-modal similarity and local cross-modal similarity respectively obtain multi-scale modal contrast loss function under coarse granularity and fine granularity, train retrieval model;Image data or text data to be searched is obtained using the trained retrieval model to obtain the retrieval result, improve retrieval speed, efficiency and retrieval quality.

Description

Technical Field

[0001] This invention relates to the field of cross-modal retrieval technology, and in particular to a text and image retrieval method and system based on coarse-grained and fine-grained modal interaction. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of the internet, various types of media data are emerging rapidly. Significant differences exist between these modalities in data representation, feature extraction methods, and data distribution, posing higher demands on cross-modal retrieval methods. Therefore, how to reduce the heterogeneity between modalities and achieve efficient and accurate cross-modal retrieval is one of the main challenges in the current research field of cross-modal retrieval.

[0004] Current research in cross-modal retrieval includes coarse-grained retrieval methods and fine-grained retrieval methods.

[0005] Coarse-grained retrieval methods typically explore the semantic correspondence between entire images and complete sentences. The most common approach usually maps image and text data into a common embedding space and then directly calculates their similarity. Early canonical correlation analysis found a linear transformation between the two modalities by maximizing pairwise correlations in the common subspace. With the development of deep learning, some methods use convolutional neural networks to extract features from images and text and map them into a semantic space for direct similarity calculation. Other methods tend to learn cross-modal representations end-to-end, such as using a two-branch neural network to learn joint embeddings of images and text, calculating cosine similarity between the global representations of the two modalities under triplet loss and proximity constraints.

[0006] Fine-grained retrieval methods decompose image and text information into local regions or subsequences for feature extraction and matching, and then synthesize local similarity to obtain the final matching result. Some researchers use cross-stacked attention to capture the correspondence between image regions and words to facilitate complete potential alignment. However, this method can only focus on one-sided image regions or words in text, and cannot symmetrically focus on the correlation between the image and text modalities. Therefore, other researchers have proposed an adaptive message passing method to achieve cross-modal interaction and extract the most salient features in text-image matching.

[0007] However, coarse-grained retrieval has limitations in capturing deeper relationships between different modalities of data. While fine-grained retrieval methods are more accurate than coarse-grained methods, they typically involve significant computation, resulting in relatively low retrieval efficiency. Therefore, it can be seen that both coarse-grained and fine-grained retrieval methods have their own limitations.

[0008] Secondly, ensuring semantic consistency between images and text in the common embedding space is crucial. Existing methods typically employ triplet ranking loss based on multimodal contrastive learning to achieve this goal. This involves adjusting the sample positions in the embedding space to bring matching image-text pairs closer together while widening the distance between mismatched image-text pairs. However, this method may result in situations where two highly similar samples of the same modality are too close together, leading to errors in the retrieval results. Summary of the Invention

[0009] To address the aforementioned issues, this invention proposes a text and image retrieval method and system based on coarse-grained and fine-grained modal interaction. The method comprises coarse-grained global retrieval and fine-grained local retrieval. Coarse-grained global retrieval utilizes coarse-grained text context features and global image features for global similarity interaction, improving retrieval speed and efficiency. Fine-grained local retrieval establishes local similarity interaction between image regions and text words through bidirectional stacked Transformers, enhancing retrieval quality.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] In a first aspect, the present invention provides a text and image retrieval method based on coarse- and fine-grained modal interaction, comprising:

[0012] Obtain sample image-text pairs, which include sample images and sample text. Extract global image features from the sample images and fine-grained word features and text context features from each word in the sample text.

[0013] Global cross-modal similarity is obtained based on the similarity between global image features and text context features;

[0014] Based on the global features of the image and the fine-grained features of each word, the region features that best match the word in the sample image are retrieved through the word tag sequence, thereby obtaining the local cross-modal similarity;

[0015] The design incorporates a multi-scale modal contrast loss function, including inter-modal contrast loss and intra-modal contrast loss. Based on global cross-modal similarity and local cross-modal similarity, coarse-grained and fine-grained multi-scale modal contrast loss functions are obtained respectively, and the constructed retrieval model is trained.

[0016] The trained retrieval model is used to obtain retrieval results for the image or text data to be retrieved.

[0017] As an alternative implementation method, the process of extracting global image features includes: after downsampling the feature map of the sample image, upsampling the downsampled feature map using a progressive upsampling method and deconvolution operation to obtain global image features.

[0018] As an alternative implementation method, the process of extracting features from sample text includes: encoding the sample text to obtain vector representations of each word, i.e., fine-grained word features; converting the word vector representations into sequence form and then extracting contextual features of forward and backward information to obtain text contextual features.

[0019] As an alternative implementation, cosine similarity is used to calculate global cross-modal similarity for global image features and text context features.

[0020] As an alternative implementation, based on global image features and fine-grained word features for each word, a stacked forward Transformer model and a backward Transformer model are used to calculate the local cross-modal similarity S. l :

[0021]

[0022] Where p(y) l |y l-1 , ..., y1, x0; θ fwd In the forward Transformer model, a marker y is generated at position l in the sample image. l The probability of ; p(y1|y2,…,y l ,x0;θ bwd ) generates a marker y at position l in the sample image in the inverse Transformer model. l The probability of θ fwd For the parameters of the forward Transformer model, θ bwd For parameters of the inverse Transformer model; y l ,…,y1 is the label sequence in the forward Transformer model, y1, y2,…,y l It is the label sequence in the inverse Transformer model, where x0 is the global feature of the image.

[0023] As an alternative implementation, the intermodal contrast loss L r for:

[0024]

[0025] Where S(·) is the similarity function; δ is the marginal hyperparameter; This is a hard negative image sample anchored to sample image I. This is a hard negative text sample anchored by the sample text T.

[0026] As an alternative implementation, the intramodal contrast loss L a for:

[0027]

[0028]

[0029]

[0030] in, The distance between each pair of hard negative samples anchored by sample text T and positive samples of the same modality; Let σ be the distance between each pair of hard negative samples anchored by sample image I and positive samples of the same modality; σ is the inter-modal safety variable. All are correction factors to dynamically adjust the distance scale between sample pairs of different modalities; S(·) is the similarity function; This represents a hard negative image sample anchored to sample image I. and its corresponding hard negative text samples This represents a hard negative image sample anchored to the sample text T. and its corresponding hard negative text samples

[0031] Secondly, the present invention provides a text and image retrieval system based on coarse- and fine-grained modal interaction, comprising:

[0032] The feature extraction module is configured to acquire sample image-text pairs, which include sample images and sample text. It extracts global image features from the sample images and fine-grained word features and text context features from each word in the sample text.

[0033] The coarse-grained global retrieval module is configured to obtain global cross-modal similarity based on the similarity between global image features and text context features.

[0034] The fine-grained local retrieval module is configured to retrieve the region features in the sample image that best match the word based on the global features of the image and the fine-grained features of each word through the word tag sequence, thereby obtaining the local cross-modal similarity.

[0035] The loss construction module is configured to design a multi-scale modal contrast loss function that includes inter-modal contrast loss and intra-modal contrast loss. Based on global cross-modal similarity and local cross-modal similarity, the multi-scale modal contrast loss function is obtained at coarse and fine granular levels, respectively, and the constructed retrieval model is trained.

[0036] The image and text retrieval module is configured to retrieve either image or text data, and uses a trained retrieval model to obtain retrieval results.

[0037] Thirdly, the present invention provides an electronic device including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the first aspect.

[0038] Fourthly, the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in the first aspect.

[0039] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0041] This invention employs an image encoder and progressive upsampling method to extract global image features, alleviating the problem of losing important semantic information during image feature sampling. For text processing, the BERT model is used to extract semantic information from the text, converting the text into word vectors that can be input into a deep neural network. In order to better capture the global features of the text, a bidirectional long short-term memory network is further used to extract text context features containing forward and backward information, thereby improving the ability to model text features.

[0042] This invention proposes a text and image retrieval method and system based on coarse-grained and fine-grained modal interaction. To more comprehensively capture the deep semantic relationships between modalities, a coarse-grained and fine-grained modal interaction method is designed, including coarse-grained global retrieval and fine-grained local retrieval. Coarse-grained global retrieval utilizes coarse-grained text context features and global image features for global similarity interaction. It can independently calculate the global features of two modalities without considering the complex interactions between the two modalities during the calculation process. Furthermore, it calculates the global features of all image and text samples before retrieval, effectively avoiding redundant feature calculations during actual retrieval and improving retrieval speed and efficiency. Fine-grained local retrieval uses stacked forward and backward Transformer models to align the local features of images and text to calculate their similarity. This captures the correlation information between modalities from two directions, thereby improving the quality of retrieval results.

[0043] To address the issue of two highly similar samples of the same modality being too close to each other, this invention designs a multi-scale modal contrast loss function to ensure that mutually matching image-text pairs are closer together, while the distance between data of the same modality is adjusted on multiple scales according to the degree of matching, with semantically more similar data being farther apart, thereby avoiding retrieval errors.

[0044] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0045] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0046] Figure 1 This is a flowchart illustrating the framework of the image and text retrieval method based on coarse-grained and fine-grained modal interaction provided in Embodiment 1 of the present invention. Detailed Implementation

[0047] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0048] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0049] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. Furthermore, it should be understood that the terms “comprising” and “including”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus.

[0050] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0051] Example 1

[0052] like Figure 1 As shown, this embodiment proposes a text and image retrieval method based on coarse-grained and fine-grained modal interaction, including:

[0053] Obtain sample image-text pairs, which include sample images and sample text. Extract global image features from the sample images and fine-grained word features and text context features from each word in the sample text.

[0054] Global cross-modal similarity is obtained based on the similarity between global image features and text context features;

[0055] Based on the global features of the image and the fine-grained features of each word, the region features that best match the word in the sample image are retrieved through the word tag sequence, thereby obtaining the local cross-modal similarity;

[0056] The design incorporates a multi-scale modal contrast loss function, including inter-modal contrast loss and intra-modal contrast loss. Based on global cross-modal similarity and local cross-modal similarity, coarse-grained and fine-grained multi-scale modal contrast loss functions are obtained respectively, and the constructed retrieval model is trained.

[0057] The trained retrieval model is used to obtain retrieval results for the image or text data to be retrieved.

[0058] In this embodiment, a sample image-text pair, including sample image and sample text, is obtained, and global features of the sample image are extracted using an image encoder and a progressive upsampling method.

[0059] Specifically, ResNet-50 is used as the image encoder. ResNet-50 is a classic convolutional neural network structure with powerful feature extraction capabilities. However, during the feature extraction process, the image undergoes a series of operations such as convolution and pooling. These downsampling operations can cause the image to lose some fine-grained visual information, which may affect the richness and representational power of the final feature representation.

[0060] Therefore, this embodiment employs a progressive upsampling method, which uses higher-resolution feature maps extracted from earlier layers as guidance to gradually restore the spatial details of the last convolutional feature map, such as... Figure 1 As shown. Taking a 224×224 input image as an example, after feature map downsampling by ResNet, a 7×7 feature map is obtained. Then, through progressive upsampling, the feature map is upsampled using deconvolution operation to expand it into a 56×56 feature map, which is the final global image feature x0.

[0061] In this embodiment, the BERT model is used as an efficient text representation method to process the sample text, extracting the semantic information of the text and converting the text into word vectors that can be input into a deep neural network. Then, in order to better capture the global features of the text, a bidirectional long short-term memory network is further used to extract text context features containing forward and backward information, thereby improving the ability to model text features.

[0062] Specifically: The input text T with L words is encoded by the trained BERT model to obtain the vector representation of each word, i.e., the fine-grained word feature y = [y1, y2, ..., y3]. l Then, the word vector representation is converted into a sequence form and input into the Bi-LSTM model to obtain the final text context features y0.

[0063] In this embodiment, a coarse-grained modal interaction method is designed, which includes coarse-grained global retrieval and fine-grained local retrieval. First, coarse-grained global retrieval is performed. Coarse-grained global retrieval refers to a method that uses only the global features of two modalities to perform cross-modal retrieval.

[0064] Specifically, the global image features x0 generated by the image encoder and the text context features y0 generated by the text encoder are considered as coarse-grained modal features, and cosine similarity is used to measure the global cross-modal similarity between two samples:

[0065] Coarse-grained global retrieval can independently compute global features for two modalities without considering the complex interactions between them during computation. Furthermore, computing global features for all image and text samples before retrieval effectively avoids redundant feature computation during the actual retrieval process. Therefore, global retrieval offers significant advantages in both speed and efficiency.

[0066] Then, fine-grained local retrieval is performed, which focuses on retrieving information by fully utilizing the local features of both modalities. This embodiment uses stacked forward and backward Transformer models to align the local features of the image and text to calculate their similarity. This allows for the capture of correlation information between modalities from both directions, thereby improving the quality of the retrieval results.

[0067] Specifically, each Transformer model consists of a Masked Multi-Head Attention layer, which allows each word vector to focus on its own information and the information of other vectors. Then, the text information and global image features enter a cross-attention layer, which enables the text to effectively focus on the global image features. Finally, there is a feedforward layer. In addition, this embodiment adds a dropout layer after each operation and wraps it in a residual connection to prevent overfitting, followed by layer normalization.

[0068] The global image feature x0 is compared with the vector representation y = [y1, y2, ..., y] of each word obtained from the BERT model. l As input, the region features that best match the word in the sample image are retrieved through the word tag sequence in the bidirectional stacked Transformer model, and finally the local cross-modal similarity is obtained:

[0069]

[0070] Where p(y) l |y l-1 , ..., y1, x0; θ fwd In the forward Transformer model, a marker y is generated at position l in the sample image. l The probability of ; p(y1|y2,…,y l ,x0;θ bwd ) generates a marker y at position l in the sample image in the inverse Transformer model. l The probability of θ fwd For the parameters of the forward Transformer model, θ bwd These are the parameters for the inverse Transformer model; the label sequence in the forward Transformer model is y. lThe labeled sequence in the inverse Transformer model is y1, y2, ..., y1. l .

[0071] Ensuring semantic consistency between images and text in a common embedding space is crucial. Previous methods typically employ triplet ranking loss based on multimodal contrastive learning to achieve this goal. This involves adjusting the sample positions in the embedding space to bring matching image-text pairs closer together while widening the distance between mismatched pairs. However, these methods may result in situations where two highly similar samples of the same modality are too close together, leading to errors in the retrieval results.

[0072] Therefore, in order to ensure the global and local semantic relevance between images and text while reducing noise interference, this embodiment proposes a multi-scale modal contrast loss function, which makes the distance between mutually matched image-text pairs closer, while the distance between data of the same modality is adjusted on multiple scales according to the degree of matching, and the more semantically similar the data, the farther apart they are.

[0073] The multi-scale modal contrast loss function is:

[0074] L mmc =L r +L a ;

[0075] Among them, L r Intermodal contrast loss is used to control the distance between sample pairs from different modalities; L a It is an intramodal contrast loss used to adjust the distance between samples of the same modality at multiple scales.

[0076] The following section provides a detailed introduction to inter-modal contrast loss and intra-modal contrast loss.

[0077] First, define (I, T) as a matched image-text pair, where I represents the image and T represents the text. Then, define two pairs of hard negative sample pairs by using either the image or text as anchor points to obtain hard negative sample pairs. and in, This indicates that hard negative image samples are found using image I as the anchor point. and its corresponding hard negative text samples Similarly, This indicates finding hard negative image samples using text T as the anchor point. and its corresponding hard negative text samples

[0078] Then, the intermodal contrast loss is defined as: ensuring that a given hard negative sample pair and The cross-modal distance between them is at least δ units greater than the cross-modal distance between positive sample pairs (I, T), specifically expressed as:

[0079]

[0080] Where S(·) is the similarity function, used to measure the similarity between two samples (as mentioned above, it can be global cross-modal similarity or local cross-modal similarity), and δ is the marginal hyperparameter.

[0081] Secondly, for intra-modal contrast loss, paired sample images and sample text are first treated as a whole, and then the multi-scale distance between different sample pairs is considered. Specifically, this is expressed as:

[0082]

[0083]

[0084]

[0085] in, Let T be the distance between the image and text hard negative sample pairs anchored by text T and the positive samples of the same modality. Let I be the distance between the hard negative sample pairs of images and text, with image I as the anchor point, and the positive samples of the same modality; σ is the inter-modal safety variable; All of these are correction factors, which are used as weights to dynamically adjust the distance scale between sample pairs of different modalities.

[0086] Substituting the global cross-modal similarity and local cross-modal similarity into the multi-scale modal contrast loss function, we obtain the coarse-grained multi-scale modal contrast loss function L. mmc (S c And fine-grained multi-scale modal contrast loss function L mmc (S f Therefore, the overall objective function of the retrieval model is: L McF =L mmc (S c )+L mmc (S f ).

[0087] To verify the effectiveness of the method in this embodiment, it was evaluated on the Flickr30K and COCO-5K datasets under the same experimental environment, and the results were compared with state-of-the-art coarse-grained and fine-grained retrieval methods. Table 1 shows the comparative experimental results on the Flickr30K dataset, Table 2 shows the comparative experimental results on the COCO-5K dataset, and Table 3 shows the ablation experimental results on the COCO-5K dataset. For convenience, in Tables 1-3, I represents the coarse-grained retrieval method, i.e., the method that uses only the globally represented image and text features for retrieval, and II represents the fine-grained retrieval method, i.e., the method that uses the local features of image regions and word tags for retrieval.

[0088] Compared to coarse-grained retrieval methods that rely solely on global features, this embodiment combines both coarse and fine-grained features, resulting in superior accuracy. Comparative experiments on two datasets demonstrate that this embodiment significantly outperforms coarse-grained retrieval methods. For fine-grained retrieval, this embodiment employs coarse- and fine-grained similarity training for CMC (Concurrent Matching Model). This method brings matching images and text in the common space closer together, while simultaneously mitigating the interference caused by closely spaced samples of the same modality, thus reducing interference from similar data. In summary, compared to simple local feature alignment, this embodiment demonstrates superior performance, exhibiting higher retrieval efficiency and accuracy.

[0089] Table 1 shows the comparative experimental results on the Flickr30K dataset.

[0090]

[0091]

[0092] Table 2 shows the comparative experimental results on the COCO-5K dataset.

[0093]

[0094] Table 3 shows the ablation experimental results on the COCO-5K dataset.

[0095]

[0096] It should be noted that all data acquisition is conducted in accordance with laws and regulations and with user consent, and the data is used legally.

[0097] Example 2

[0098] This embodiment provides a text and image retrieval system based on coarse-grained and fine-grained modal interaction, including:

[0099] The feature extraction module is configured to acquire sample image-text pairs, which include sample images and sample text. It extracts global image features from the sample images and fine-grained word features and text context features from each word in the sample text.

[0100] The coarse-grained global retrieval module is configured to obtain global cross-modal similarity based on the similarity between global image features and text context features.

[0101] The fine-grained local retrieval module is configured to retrieve the region features in the sample image that best match the word based on the global features of the image and the fine-grained features of each word through the word tag sequence, thereby obtaining the local cross-modal similarity.

[0102] The loss construction module is configured to design a multi-scale modal contrast loss function that includes inter-modal contrast loss and intra-modal contrast loss. Based on global cross-modal similarity and local cross-modal similarity, the multi-scale modal contrast loss function is obtained at coarse and fine granular levels, respectively, and the constructed retrieval model is trained.

[0103] The image and text retrieval module is configured to retrieve either image or text data, and uses a trained retrieval model to obtain retrieval results.

[0104] It should be noted that the above modules correspond to the steps described in Embodiment 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0105] In further embodiments, the following is also provided:

[0106] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in Embodiment 1. For brevity, further details are omitted here.

[0107] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0108] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0109] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in Embodiment 1.

[0110] The method in Example 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0111] A computer program product includes a computer program that, when executed by a processor, implements the method described in Embodiment 1.

[0112] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0113] The computer program code used to implement the methods of the present invention may be written in one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the computer or other programmable data processing device, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0114] In the context of this invention, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and so on. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0115] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0116] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A text and image retrieval method based on coarse-grained and fine-grained modal interaction, characterized in that, Including: Obtain sample image-text pairs, where the sample image-text pairs include sample images and sample texts. Extract global image features from the sample images, and extract word fine-grained features and text context features for each word in the sample texts. Obtain the global cross-modal similarity based on the similarity between the global image features and the text context features. Based on the global image features and the word fine-grained features of each word, retrieve the region features in the sample image that best match the word through a word token sequence, thereby obtaining the local cross-modal similarity. Specifically, based on global image features and fine-grained word features for each word, stacked forward and backward Transformer models are used to calculate local cross-modal similarity. : ； in, For the forward Transformer model in the sample image Location generation marker The probability of; In the inverse Transformer model, in the sample image Location generation marker The probability of; For parameters of the forward Transformer model, These are the parameters for the inverse Transformer model; It is the labeled sequence in the forward Transformer model. It is the labeled sequence in the inverse Transformer model. For global features of the image; Design a multi-scale modal contrast loss function including inter-modal contrast loss and intra-modal contrast loss. Thus, obtain the multi-scale modal contrast loss functions at coarse-grained and fine-grained levels respectively based on the global cross-modal similarity and the local cross-modal similarity, and train the constructed retrieval model. Intermodal contrast loss for: ； in, It is the similarity function; δ is the marginal hyperparameter; This is a hard negative image sample anchored to sample image I. This is a hard negative text sample anchored to sample text T; The intramodal contrast loss for: ； ； ； in, The distance between each pair of hard negative samples anchored by sample text T and positive samples of the same modality; The distance between each pair of hard negative samples anchored by sample image I and a positive sample of the same modality; For intermodal safety variables; , These are all correction factors to dynamically adjust the distance scale between sample pairs of different modalities; This represents a hard negative image sample anchored to sample image I. and its corresponding hard negative text samples , This represents a hard negative image sample anchored to the sample text T. and its corresponding hard negative text samples ; For the image data or text data to be retrieved, use the trained retrieval model to obtain the retrieval result.

2. A method for image-text retrieval based on coarse-grained and fine-grained modal interaction according to claim 1, wherein The process of extracting global image features includes: after downsampling the feature map of the sample image, use the progressive upsampling method to upsample the feature map obtained by downsampling through deconvolution operations to obtain the global image features. The process of extracting features in the sample text includes: encoding the sample text to obtain the vector representation of each word, that is, the word fine-grained features, convert the word vector representation into a sequence form, and then extract the context features of forward and backward information to obtain the text context features.

3. The image and text retrieval method based on coarse-grained and fine-grained modal interaction as described in claim 1, characterized in that, Use cosine similarity to calculate the global cross-modal similarity between the global image features and the text context features.

4. A text and image retrieval system based on coarse-grained and fine-grained modal interaction, characterized in that, Including: A feature extraction module configured to obtain sample image-text pairs, where the sample image-text pairs include sample images and sample texts, extract global image features from the sample images, and extract word fine-grained features and text context features for each word in the sample texts. A coarse-grained global retrieval module configured to obtain the global cross-modal similarity based on the similarity between the global image features and the text context features. A fine-grained local retrieval module configured to retrieve the region features in the sample image that best match the word through a word token sequence based on the global image features and the word fine-grained features of each word, thereby obtaining the local cross-modal similarity. Specifically, based on global image features and fine-grained word features for each word, stacked forward and backward Transformer models are used to calculate local cross-modal similarity. : ； in, For the forward Transformer model in the sample image Location generation marker The probability of; In the inverse Transformer model, in the sample image Location generation marker The probability of; For parameters of the forward Transformer model, These are the parameters for the inverse Transformer model; It is the labeled sequence in the forward Transformer model. It is the labeled sequence in the inverse Transformer model. For global features of the image; A loss construction module configured to design a multi-scale modal contrast loss function including inter-modal contrast loss and intra-modal contrast loss. Thus, obtain the multi-scale modal contrast loss functions at coarse-grained and fine-grained levels respectively based on the global cross-modal similarity and the local cross-modal similarity, and train the constructed retrieval model. Intermodal contrast loss for: ； in, It is the similarity function; δ is the marginal hyperparameter; This is a hard negative image sample anchored to sample image I. This is a hard negative text sample anchored to sample text T; The intramodal contrast loss for: ； ； ； in, The distance between each pair of hard negative samples anchored by sample text T and positive samples of the same modality; The distance between each pair of hard negative samples anchored by sample image I and a positive sample of the same modality; For intermodal safety variables; , These are all correction factors to dynamically adjust the distance scale between sample pairs of different modalities; This represents a hard negative image sample anchored to sample image I. and its corresponding hard negative text samples , This represents a hard negative image sample anchored to the sample text T. and its corresponding hard negative text samples ; An image-text retrieval module configured to use the trained retrieval model for the image data or text data to be retrieved to obtain the retrieval result.

5. An electronic device, characterized in that, Including a memory, a processor, and computer instructions stored on the memory and running on the processor. When the computer instructions are run by the processor, the method according to any one of claims 1-3 is completed.

6. A computer-readable storage medium, characterized in that, For storing computer instructions, when the computer instructions are executed by the processor, the method according to any one of claims 1-3 is completed.

7. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the method described in any one of claims 1-3.

Images