A cross-modal semantic awareness-based image and text retrieval method and system

Abstract

This invention proposes a cross-modal semantic perception-based image and text retrieval method and system, relating to the field of image and text retrieval technology. The specific scheme includes: acquiring images and text to be matched; inputting the images and text into a trained cross-modal semantic perception network to obtain the similarity between the images and text; matching the images and text based on the similarity to obtain the image and text retrieval results; wherein the cross-modal semantic perception network extracts text features and image features from the text and images respectively, and performs binary relation reasoning and multi-variable relation reasoning on the fused text features and image features respectively to obtain the final image features, and calculates the similarity between the final image features and text features; this invention, based on a multi-level information dynamic fusion module and a relation perception module, deeply explores the semantic association between vision and language, and provides a more accurate estimation of the similarity between images and text.

Description

Technical Field

[0001] This invention belongs to the field of image and text retrieval technology, and in particular relates to an image and text retrieval method and system based on cross-modal semantic perception. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Image-text retrieval tasks include two subtasks: image-text search (IR) and text-image search (TR), aiming to achieve better understanding and alignment through the fusion of information from different modalities. Current image-text retrieval technologies can be broadly classified into two categories: coarse-grained matching and fine-grained matching. Coarse-grained matching technology captures global semantic relationships by mapping images and text into a shared embedding space.

[0004] While coarse-grained matching methods are more intuitive and concise, they often overlook local details in images and text. With technological advancements, fine-grained matching methods have become the mainstream research approach. Existing fine-grained matching techniques, although effective in constructing local semantic connections between images and text, can still be hampered by redundant information and noise in images, as well as semantically meaningless words in text, all of which can interfere with global matching. A single text description often fails to encompass all details and objects in an image; for example, it may lack words corresponding to specific regions of the image or contain semantically meaningless words, such as "A" and "the" in "A woman is walking a dog on the beach." These can affect the accuracy of feature matching between images and text. Furthermore, in the retrieval process, in addition to considering entity alignment between image regions and text words, semantic consistency between different modal relationships is also crucial. These issues result in low accuracy for image-text retrieval using existing solutions. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, this invention provides a cross-modal semantic perception-based image and text retrieval method and system. Based on a multi-level information dynamic fusion module and a relationship perception module, it deeply explores the semantic association between vision and language, and makes a more accurate estimate of the similarity between images and text.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0007] The first aspect of this invention provides a text and image retrieval method based on cross-modal semantic awareness.

[0008] A cross-modal semantic awareness-based image and text retrieval method includes:

[0009] Get the image and text to be matched;

[0010] Images and text are input into a pre-trained cross-modal semantic perception network to obtain the similarity between the images and text;

[0011] Image and text are matched based on similarity to obtain image and text retrieval results;

[0012] The cross-modal semantic perception network extracts text features and image features from text and images respectively, and performs binary relation reasoning and multi-relation reasoning on the fused text features and image features respectively to obtain the final image features, and calculates the similarity between the final image features and text features.

[0013] Furthermore, the cross-modal semantic perception network includes a text feature extraction module, an image feature extraction module, a relationship perception module, and a similarity calculation module;

[0014] The text feature extraction module is used to extract text features.

[0015] The image feature extraction module is used to extract image features;

[0016] The relation-aware module is used to perform binary relation reasoning and multi-variable relation reasoning on the fused text features and image features respectively to obtain the final image features;

[0017] The similarity calculation module is used to calculate the final similarity between image features and text features.

[0018] Furthermore, the extraction of the text features specifically includes:

[0019] LSTM is used to generate initial features at the word level;

[0020] Initial features at the word level are enhanced using a graph convolutional network to obtain local features;

[0021] Text features are obtained by performing global semantic reasoning on local features through a bidirectional gated loop unit.

[0022] Furthermore, the extraction of the image features specifically includes:

[0023] Pre-trained Faster-RCNN is used to generate initial features at the region level;

[0024] Initial features at the region level are enhanced using graph convolutional networks and residual connections to obtain local features;

[0025] By using a gated loop unit, global semantic reasoning is performed on local features to obtain global features;

[0026] The multi-level information dynamic fusion module performs multi-level mixing of local and global image features to obtain image features.

[0027] Furthermore, the step of performing binary relation reasoning and multivariate relation reasoning on the fused text features and image features respectively involves inputting the fused text features and image features into two independent convolutional streams to perform binary relation reasoning and multivariate relation reasoning, obtaining a binary relation matrix and a multivariate relation matrix, and then weighted and fused the two matrices to obtain the final image features.

[0028] Furthermore, the fused text features and image features, before performing binary relation reasoning and multi-relation reasoning, require the following processing:

[0029] The fused features are mapped to a low-dimensional space, then the dimensions of the low-dimensional features are expanded, and finally added to their transpose to obtain a pairwise combination of all image region representations.

[0030] Furthermore, the cross-modal semantic perception network is trained using a triplet loss function to minimize the similarity between positive samples and maximize the similarity between negative samples.

[0031] A second aspect of the present invention provides a text and image retrieval system based on cross-modal semantic awareness.

[0032] A cross-modal semantic awareness-based image and text retrieval system includes an image and text acquisition module, a similarity calculation module, and an image and text matching module.

[0033] The image and text acquisition module is configured to acquire the image and text to be matched.

[0034] The similarity calculation module is configured to input images and text into a trained cross-modal semantic perception network to obtain the similarity between the images and text.

[0035] The image-text matching module is configured to match images and text based on similarity to obtain image-text retrieval results.

[0036] The cross-modal semantic perception network extracts text features and image features from text and images respectively, and performs binary relation reasoning and multi-relation reasoning on the fused text features and image features respectively to obtain the final image features, and calculates the similarity between the final image features and text features.

[0037] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of a cross-modal semantic awareness-based image and text retrieval method as described in the first aspect of the present invention.

[0038] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a cross-modal semantic awareness-based image and text retrieval method as described in the first aspect of the present invention.

[0039] The above one or more technical solutions have the following beneficial effects:

[0040] This invention proposes a cross-modal semantic perception network, CSA-Net, which introduces a multi-level information dynamic fusion module to correct and enhance the local and global features of an image, thereby obtaining a more accurate visual representation. At the same time, a relation-aware module (RAM) is also designed to use text information to identify binary and multi-relationships between objects in an image, which helps the model to understand the background and overall context of the image more accurately.

[0041] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0042] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0043] Figure 1 This is a flowchart of the method in the first embodiment.

[0044] Figure 2 This is an example diagram of a scenario from the first embodiment.

[0045] Figure 3 This is a diagram of the cross-modal semantic perception network structure of the first embodiment.

[0046] Figure 4 This is a structural diagram of the multi-level information dynamic fusion module in the first embodiment.

[0047] Figure 5 This is a structural diagram of SA and GA in the multi-level information dynamic fusion module of the first embodiment.

[0048] Figure 6 This is a structural diagram of the relation-aware module in the first embodiment. Detailed Implementation

[0049] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0050] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0051] Among numerous tasks in computer vision, graph-based reasoning techniques have proven to be an efficient method for understanding and processing relationships between visual elements. By constructing graph structures to represent objects in an image and their relationships, it provides an intuitive way for models to capture the spatial layout of an image and the interactions between objects. Based on graph-based reasoning techniques, this invention proposes a cross-modal semantic awareness network, CSA-Net, which aims to deeply explore the semantic connections between vision and language. This network first uses a graph convolutional network (GCN) to reason about the extracted initial features to gain a deeper understanding of image and text data. Then, by introducing a multi-level information dynamic fusion module into the image feature extraction module, local and global image features are corrected and enhanced to obtain a more accurate visual representation. At the same time, a relation-aware module (RAM) is also designed, which uses text information to identify binary and multivariate relationships between objects in the image, helping the model to more accurately understand the background and overall context of the image.

[0052] Example 1

[0053] One embodiment of this disclosure provides a text-image retrieval method based on cross-modal semantic awareness, such as... Figure 1 As shown, it includes the following steps:

[0054] Step S1: Obtain the image and text to be matched.

[0055] In the image-text retrieval of this embodiment, as shown in scenario 2, if image-text matching is performed using the similarity between feature vectors, the matched images and texts are used as the retrieval result set. Therefore, the images and texts to be matched are obtained first.

[0056] Step S2: Input the image and text into the trained cross-modal semantic perception network to obtain the similarity between the image and the text.

[0057] The cross-modal semantic perception network extracts text features and image features from text and images respectively, and performs binary relation reasoning and multi-relation reasoning on the fused text features and image features respectively to obtain the final image features, and calculates the similarity between the final image features and text features.

[0058] The cross-modal semantic awareness network, such as Figure 3 As shown, it includes a text feature extraction module, an image feature extraction module, a relationship awareness module, and a similarity calculation module. The following is a detailed description of each model.

[0059] 1. The text feature extraction module is used to extract text features;

[0060] Furthermore, the extraction of the text features specifically includes:

[0061] LSTM is used to generate initial features at the word level;

[0062] Initial features at the word level are enhanced using a graph convolutional network to obtain local features;

[0063] Text features are obtained by performing global semantic reasoning on local features through a bidirectional gated loop unit.

[0064] Specifically, firstly, LSTM is used to capture the sequence dependencies of words or tags in the input sentence, resulting in initial word-level features:

[0065] T={t j |j = 1, ..., M, t j ∈R d2} (1)

[0066] Where T is the initial feature, t j d2 represents the embedding vector of each word, M is the number of words, and d2 represents the dimension of the word features.

[0067] Then, the extracted text features are input into a graph convolutional network (GCN) to obtain semantically enhanced local features:

[0068]

[0069] Finally, due to the strong contextual relationships between words in the text, a bidirectional GRU (Bi-GRU) is used to learn the word vector encoding of the entire sentence information, enabling global semantic inference based on local features:

[0070]

[0071] Where, j th positive and backward Hidden states, respectively and Therefore, the eigenvector can be represented as:

[0072]

[0073] The encoded features are concatenated, and the fused feature vector is processed through a fully connected layer to obtain a global text feature E.

[0074] 2. The image feature extraction module is used to extract image features;

[0075] Furthermore, the extraction of the image features specifically includes:

[0076] Pre-trained Faster-RCNN is used to generate initial features at the region level;

[0077] Initial features at the region level are enhanced using graph convolutional networks and residual connections to obtain local features;

[0078] By using a gated loop unit, global semantic reasoning is performed on local features to obtain global features;

[0079] The multi-level information dynamic fusion module performs multi-level mixing of local and global image features to obtain image features.

[0080] Specifically, given image I, a pre-trained Faster-RCNN is used to generate initial features at the region level, and the initial features are defined as follows:

[0081] V = {V i |i = 1, 2, ..., n, Vi ∈ R d1} (5)

[0082] Where n represents the number of regions detected in the image, and d1 is the feature dimension of a single image.

[0083] After the initial features are processed by the Graph Convolutional Network (GCN) and Gated Recurrent Unit (GRU), an understanding of the global semantics of the image is obtained. However, this method does not distinguish between key targets and redundant information in the image. To solve this problem, a local feature branch of the image is added to obtain the relationship between the local and global features of the image.

[0084] Specifically, the semantic relevance between regions is enhanced by calculating the pairwise affinity of these regions in the embedding space; then, by combining a graph convolutional network (GCN) and residual connections, computation is performed on the constructed fully connected graph to enhance the representation of image regions and obtain local features.

[0085] V l =W r (RVW g )+V (6)

[0086] Where Wg is the weight matrix of the D×D GCN layer, R is the affinity matrix of shape k×k, and the affinity matrix R is normalized row by row according to the routine, outputting It is an enhanced representation of the relationships between nodes in an image region.

[0087] To further perform global semantic reasoning, a GRU is used to obtain a description of the entire scene, resulting in global features:

[0088] V g =GRU(V * (7)

[0089] Global features tend to capture the overall meaning and background of an image, while local features focus on describing specific elements or regions within the image. Global features become particularly crucial when there are only a few objects in an image or when the main content is difficult to identify clearly using local features. Conversely, the use of local features is especially important when an image contains multiple salient objects, each of which is noteworthy. While manually adjusting weights can improve retrieval accuracy, this method is both time-consuming and laborious. If the model could automatically generate weights based on global and local features, it would significantly improve the visual performance of the image. To address these issues, a multi-level information dynamic fusion module is introduced, such as... Figure 3 As shown, this module optimizes image feature representation by fusing multi-level features and dynamically generates weights to achieve better visual expression.

[0090] Specifically, the overall structure of the multi-level information dynamic fusion module is as follows: Figure 4 As shown, we will first explain the self-attention (SA) and guided attention (GA) mechanisms, such as... Figure 5 As shown, self-attention (SA) and guided attention (GA) are used to optimize the generation of multi-level information. Self-attention (SA) discovers the correlation between features by performing internal similarity calculation on the input feature X, denoted by SA(x). GA(x,y) represents the guided attention mechanism, which outputs a weighted feature Z based on the similarity between feature X and feature Y under the guidance of feature Y.

[0091] like Figure 4 As shown, based on the SA and GA mentioned above, the global feature v g and local features v l The weighted global features v are obtained by inputting them into the SA and GA structures respectively. g SGA and local features v l SGA This can be expressed as a formula:

[0092]

[0093] Among them, v g SA and v l SA This represents the global and local features obtained after processing by the SA module, v g SGA and v l SGA This represents the weighted global and local features.

[0094] In addition, local features are used to correct global features, while global features supplement local features.

[0095]

[0096] in, This represents the dot product operation between two vectors. v′ g and v′ l These represent the representations after the interaction of global and local feature information, respectively. Formula (9) is used to generate a mask using local features. This mask is used to filter and adjust global features, while global features also directly enhance the representation of local features.

[0097] Although the expressiveness of features is improved after multi-level feature interaction, multi-level mixing of image features is still required to obtain a fused image feature representation. Therefore, we superimpose global and local features and perform a linear transformation to obtain learnable dynamic weights γ1 and γ2, which are used to generate a fused image feature R, expressed by the formula:

[0098]

[0099] Third, the relation-aware module is used to perform binary relation reasoning and multivariate relation reasoning on the fused text features and image features respectively to obtain the final image features;

[0100] While fused image features enhance the understanding of image relationships, combining them with text features is essential for a better understanding of the semantic relationships between images and text. Therefore, a relationship-aware module was designed to explore the correlations between images and text, such as... Figure 5 As shown, binary relation reasoning and multivariate relation reasoning are performed through two independent convolutional streams, which are the binary relation reasoning module and the multivariate relation reasoning module, respectively.

[0101] The dynamically fused image features R and text features E are mapped to a low-dimensional space to obtain low-dimensional features. low-dimensional features Expanding the dimension of the image region and then adding it to its transpose yields a pairwise combination of all image region representations, expressed by the formula:

[0102]

[0103] in, Indicated in the visual region A one-dimensional vector is inserted at the first position, and repeatK() indicates that the insertion operation is repeated K times. Finally, a pairwise combination of all image region representations is obtained, denoted as R. f ∈R KXKXds .

[0104] Binary relation reasoning refers to a method used to analyze the relationships between pairs of objects or elements in an image. This reasoning method focuses on understanding the interactions, spatial relationships, and attribute relationships between two elements in an image. Since there may be multiple objects in an image, binary relation reasoning can help identify specific relationships between these objects. For example, when identifying an image of a "dog," the model needs to infer that the "dog" is the subject, and the "ball" may be a toy of the "dog," thus being related to the "dog." By understanding and utilizing these relationships, image retrieval systems can better understand the relationships between text and images and perform more accurate retrieval accordingly.

[0105] Specifically, the binary relation reasoning module uses three consecutive 1×1 convolutional layers, each followed by a ReLU activation function. The number of output channels for these three convolutional layers are ds, ... The number of convolution kernels are respectively 1. Combine pairs R f The input is fed into the binary relation reasoning module, and the output is generated after convolution. Since the correlation between the two regions is the same, therefore... Adding it to its transpose yields a symmetric matrix; finally, the softmax function is used to generate a symmetric matrix R of a binary relation. b The specific calculation formula is as follows:

[0106]

[0107] To capture the diverse relationships between visual regions, dilated convolution is used to expand the receptive field of the convolution kernel, thereby exploring these diverse relationships, such as... Figure 6 The yellow background shows the paired combinations R output by the feature fusion module. f It is input into the multi-relational reasoning module.

[0108] The multivariate relation reasoning module employs three consecutive 3×3 extended convolutional layers, each followed by a ReLU activation function; the expansion factors of these three extended convolutional layers are 1, 2, and 4, respectively. The output of the multivariate relation reasoning is obtained after the last convolutional layer and the ReLU activation layer. Similar to binary relational reasoning, Adding it to its transpose yields a symmetric matrix; finally, softmax is used to generate multivariate relations, calculated as follows:

[0109]

[0110] Based on the above binary relation reasoning and multivariate relation reasoning, image features can be represented as:

[0111]

[0112] in, and This represents the relationship between visual regions i and j.

[0113] IV. The similarity calculation module is used to calculate the final similarity between image features and text features.

[0114] Specifically, similarity is calculated using a loss function, with the following formula:

[0115]

[0116] in, It is a learned representation of two samples with different modalities; for example, X is the output feature of a text encoder for one sample, and Y is the output feature of an image encoder for another sample; where X O and Y O Cosine similarity represents the global features of two samples in a common feature space; it is used to measure the similarity between two samples.

[0117] Step S3: Match images and text based on similarity to obtain image-text retrieval results.

[0118] The triplet loss function is used to optimize the cross-modal semantic perception network. The goal of this loss function is to minimize the similarity between positive samples and maximize the similarity between negative samples. The formula is as follows:

[0119]

[0120] Where α represents the minimum boundary, [x]+≡max(x,0), (V,T) is a pair of samples, and T -There is no text paired with image V, and V - It is an image that is not paired with text T.

[0121] This embodiment uses two publicly available benchmark datasets, Flickr30K and MSCOCO, for training and evaluation of the image-text retrieval task.

[0122] Flickr30K contains 31,000 images, of which 29,000 are used for training, 1,000 for validation, and 1,000 for testing. MSCOCO contains 123,287 images, which are divided into 113,287 for training, 5,000 for validation, and 5,000 for testing. It can be observed that compared to the models in this embodiment, the proposed method achieves better results on both Flickr30K and MSCOCO, and the experimental results confirm the effectiveness of cross-modal semantic awareness networks.

[0123] Example 2

[0124] One embodiment of this disclosure provides an image-text retrieval system based on cross-modal semantic awareness, including an image-text acquisition module, a similarity calculation module, and an image-text matching module:

[0125] The image and text acquisition module is configured to acquire the image and text to be matched.

[0126] The similarity calculation module is configured to input images and text into a trained cross-modal semantic perception network to obtain the similarity between the images and text.

[0127] The image-text matching module is configured to match images and text based on similarity to obtain image-text retrieval results.

[0128] The cross-modal semantic perception network extracts text features and image features from text and images respectively, and performs binary relation reasoning and multi-relation reasoning on the fused text features and image features respectively to obtain the final image features, and calculates the similarity between the final image features and text features.

[0129] Example 3

[0130] The purpose of this embodiment is to provide a computer-readable storage medium.

[0131] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a cross-modal semantic awareness-based image and text retrieval method as described in Embodiment 1 of this disclosure.

[0132] Example 4

[0133] The purpose of this embodiment is to provide an electronic device.

[0134] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in a cross-modal semantic awareness-based image and text retrieval method as described in Embodiment 1 of this disclosure.

[0135] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A text-image retrieval method based on cross-modal semantic awareness, characterized in that, include: Get the image and text to be matched; Images and text are input into a pre-trained cross-modal semantic perception network to obtain the similarity between the images and text; Image and text are matched based on similarity to obtain image and text retrieval results; The cross-modal semantic perception network extracts text features and image features from text and images respectively, and performs binary relation reasoning and multi-relation reasoning on the fused text features and image features respectively to obtain the final image features, and calculates the similarity between the final image features and text features; The cross-modal semantic perception network includes a text feature extraction module, an image feature extraction module, a relationship perception module, and a similarity calculation module; The text feature extraction module is used to extract text features. The image feature extraction module is used to extract image features; The relation-aware module is used to perform binary relation reasoning and multi-variable relation reasoning on the fused text features and image features respectively to obtain the final image features; The similarity calculation module is used to calculate the final similarity between image features and text features; The step of performing binary relation reasoning and multivariate relation reasoning on the fused text features and image features respectively involves inputting the fused text features and image features into two independent convolutional streams, performing binary relation reasoning and multivariate relation reasoning to obtain binary relation matrices and multivariate relation matrices, and then weighted and fused the two matrices to obtain the final image features.

2. The image and text retrieval method based on cross-modal semantic awareness as described in claim 1, characterized in that, The extraction of the text features specifically involves: LSTM is used to generate initial features at the word level; Initial features at the word level are enhanced using a graph convolutional network to obtain local features; Text features are obtained by performing global semantic reasoning on local features through a bidirectional gated loop unit.

3. The image and text retrieval method based on cross-modal semantic awareness as described in claim 1, characterized in that, The extraction of the image features specifically involves: Pre-trained Faster-RCNN is used to generate initial features at the region level; Initial features at the region level are enhanced using graph convolutional networks and residual connections to obtain local features; By using a gated loop unit, global semantic reasoning is performed on local features to obtain global features; The multi-level information dynamic fusion module performs multi-level mixing of local and global image features to obtain image features.

4. The image and text retrieval method based on cross-modal semantic awareness as described in claim 1, characterized in that, The fused text and image features, before being used for binary and multi-ary relation reasoning, require the following processing: The fused features are mapped to a low-dimensional space, then the dimensions of the low-dimensional features are expanded, and finally added to their transpose to obtain a pairwise combination of all image region representations.

5. The image and text retrieval method based on cross-modal semantic awareness as described in claim 1, characterized in that, The cross-modal semantic perception network is trained using a triplet loss function to minimize the similarity between positive samples and maximize the similarity between negative samples.

6. A text and image retrieval system based on cross-modal semantic awareness, characterized in that, The image-text retrieval method as described in any one of claims 1-5 includes an image-text acquisition module, a similarity calculation module, and an image-text matching module. The image and text acquisition module is configured to acquire the image and text to be matched. The similarity calculation module is configured to input images and text into a trained cross-modal semantic perception network to obtain the similarity between the images and text. The image-text matching module is configured to match images and text based on similarity to obtain image-text retrieval results. The cross-modal semantic perception network extracts text features and image features from text and images respectively, and performs binary relation reasoning and multi-relation reasoning on the fused text features and image features respectively to obtain the final image features, and calculates the similarity between the final image features and text features.

7. An electronic device, characterized in that it comprises: Memory is used to store computer-readable instructions in a non-transitory manner. Processor, for executing the computer-readable instructions; When the computer-readable instructions are executed by the processor, they perform the method described in any one of claims 1-5.

8. A storage medium, characterized in that, The computer-readable instructions are stored non-temporarily, wherein when the computer-readable instructions are executed by a computer, the method described in any one of claims 1-5 is performed.

Images