Image-text retrieval method and system based on context-guided multimodal association

Abstract

The disclosure provides a context-guided multimodal association-based image-text retrieval method and system, and relates to the technical field of cross-modal image-text mutual retrieval, which comprises the following steps: acquiring fine-grained feature sequences of different modal images and text data; constructing a context-guided multimodal association learning network to respectively acquire regionally spatially enhanced visual context perception representations of image modal and word temporally enhanced text context perception representations of text modal; constructing a context-guided multimodal association three-branch to perform cross-modal similarity learning on context perception representations of different semantic levels, learning cross-modal association by using a vector-type similarity function, designing an objective function, and respectively realizing complementation of different semantic levels within a mode and semantic accurate alignment between different modal data based on a joint learning semantic consistency loss function and a cross-modal matching loss function; and the disclosure can realize accurate alignment between different modal.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of cross-modal image-text retrieval, in particular to an image-text retrieval method and system based on context-guided multi-modal correlation. BACKGROUND

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute the prior art.

[0003] In recent years, with the explosive growth of multimedia data on the Internet, the demand for efficient and accurate information retrieval from these data has also increased. In order to achieve more efficient and effective information retrieval, cross-modal retrieval is proposed to search for relevant data between different modalities. The importance of cross-modal retrieval research lies in its ability to bridge the semantic gap between different modalities. Vision and language are two important media for human understanding of the real world, and image-text retrieval has become an important branch of cross-modal retrieval. Existing methods have conducted a lot of research to connect visual and language modalities, and image-text retrieval, which aims to search for images related to a given query text or retrieve texts related to a given query image, has become an important research field in computer vision and natural language processing.

[0004] The main challenge of image-text retrieval is the "heterogeneous gap", which refers to the inconsistency of representation and distribution between different modalities (such as images and texts). In order to bridge this gap, early image-text retrieval research mainly focuses on mapping image and text features to a common embedding space at the global level.

[0005] However, such methods mainly focus on global-level instances, ignoring detailed semantic information existing in local-level segments. In order to solve this limitation and capture semantic information more comprehensively, recent research has begun to shift attention to local-level fine-grained segments. Specifically, fine-grained segments of images can be obtained by uniform blocking or salient object detection, while fine-grained segments of texts can be generated by cutting texts into several sentences or words. Although image-text retrieval methods that utilize local-level segments have the potential to align fine-grained segments between modalities, most methods may lack the ability to effectively mine intra-modal context relationships, such as spatial dependencies between regions in images and sequential dependencies between words in texts. Therefore, these methods often struggle to break through performance bottlenecks.

[0006] To improve the performance of image-text similarity learning, it is crucial to effectively integrate the global instances and local segments of images and texts in a unified framework that considers the contextual relationships. By adopting this framework, the similarity between images and texts can be effectively learned, providing more accurate results for image-text retrieval. However, despite the progress made in image-text retrieval technology, this task remains challenging due to the following issues:

[0007] 1) Existing methods ignore intra-modal reasoning, failing to utilize information within a single modality (image or text) to understand and retrieve relevant information.

[0008] 2) Existing methods ignore inter-modal alignment, failing to mine complementary information from each modality and use it to improve overall understanding of the data. SUMMARY

[0009] To address the above problems, the present disclosure proposes an image-text retrieval method and system based on context-guided multi-modal correlation learning, which proposes a context-guided multi-modal correlation learning network (CMCL) that consists of three stages: perception, reasoning, and alignment. It simultaneously mines intra-modal and inter-modal correlation relationships to more accurately learn cross-modal similarity, and effectively mines and fuses the complementarity between different modalities at the global and local levels to learn precise multi-modal alignment.

[0010] According to some embodiments, the present disclosure adopts the following technical solutions:

[0011] The image-text retrieval method based on context-guided multi-modal correlation includes:

[0012] Obtain image and text data of different modalities, and extract fine-grained feature sequences of image and text data of different modalities respectively; construct a context-guided multi-modal correlation learning network framework, and use fine-grained feature sequences of different modalities to capture complementary semantic relationships within modalities based on self-attention mechanism and gating mechanism, to obtain regionally spatially enhanced visual context perception representation of image modality and sequentially enhanced text context perception representation of text modality respectively;

[0013] The global inter-modal correlation mining branch, the local inter-modal correlation mining branch, and the local intra-modal correlation mining branch are designed in the context-guided multi-modal correlation learning network framework; in the global inter-modal correlation mining branch, cross-modal similarity learning is performed on the global level feature vectors of the image and the text; in the local inter-modal correlation mining branch, the cross-attention mechanism is used to fully capture the multi-modal correlation between the image regions and the text words, and the similarity between the image and the text is learned at the local level; in the local intra-modal correlation mining branch, a graph model is constructed, and the graph convolution network is used to transmit and update information between nodes in the graph, and the GRU network model is used to integrate and infer the relationship between different modalities; and the cross-modal consistency term and the cross-modal alignment term are proposed in the hybrid loss function to realize effective alignment between different modalities.

[0014] According to some embodiments, the present disclosure adopts the technical scheme as follows:

[0015] The image-text retrieval system based on context-guided multi-modal correlation comprises:

[0016] The modal data representation module acquires image and text data of different modalities, respectively extracts fine-grained feature sequences of the image and the text data of different modalities, constructs a context-guided multi-modal correlation learning network framework, and uses the fine-grained feature sequences of different modalities to capture the complementary semantic relationship within the modalities based on the self-attention mechanism and the gating mechanism, and respectively acquires the regionally spatially enhanced visual context perception representation of the image modality and the sequentially enhanced text context perception representation of the text modality;

[0017] The three-branch multi-modal correlation relationship mining module is used to design the global inter-modal correlation mining branch, the local inter-modal correlation mining branch, and the local intra-modal correlation mining branch in the context-guided multi-modal correlation learning network framework; in the global inter-modal correlation mining branch, cross-modal similarity learning is performed on the global level vectors of the image and the text; in the local inter-modal correlation mining branch, the cross-attention mechanism is used to fully capture the multi-modal correlation between the image regions and the text words, and the similarity between the image and the text is learned at the local level; in the local intra-modal correlation mining branch, a graph model is constructed, and the graph convolution network is used to transmit and update information between nodes in the graph, and the GRU network model is used to integrate and infer the relationship between different modalities;

[0018] The alignment module is used to propose the cross-modal consistency term and the cross-modal alignment term in the hybrid loss function to realize effective alignment between different modalities.

[0019] Compared with the prior art, the present disclosure has the beneficial effects that:

[0020] The disclosure proposes a context-guided multi-modal correlation learning (CMCL) framework, which simultaneously solves the intra-modal reasoning and inter-modal alignment problems in image-text retrieval. CMCL follows the key idea of "perception, reasoning, and alignment", explores and integrates intra-modal and inter-modal relationships to learn more accurate image-text similarity. Self-attention and gating mechanisms are introduced to adaptively learn the fine-grained embedding of the context perception of each modality. Then, the intra-modal relationship is deeply mined to establish connections, reasoning and correlation between different information within the same modality. In order to fully capture the correlation between different modalities, more complete inter-modal alignment is learned from the global and local levels respectively. In addition, by optimizing the hybrid loss containing the cross-modal consistency term and the cross-modal alignment term, the image-text similarity is effectively integrated, and two goals are achieved: effective complement of different types of cross-modal similarity and accurate alignment of different modalities.

[0021] The disclosure is compared with existing methods on two benchmark datasets of Flickr 30K and MS-COCO, and the solution proposed by the disclosure is proved to be effective, and the ablation experiment further proves the effectiveness of each key module in the network. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which form a part of the disclosure, are intended to provide further understanding of the disclosure, and the schematic embodiments of the disclosure and the description thereof are used to explain the disclosure, and do not constitute improper limitations on the disclosure.

[0023] Figure 1 The overall framework diagram of the context-guided multi-modal correlation learning of the embodiments of the disclosure;

[0024] Figure 2 The structure diagram of the context perception unit of the embodiments of the disclosure;

[0025] Figure 3 The influence of vector dimension on cross-modal retrieval performance on Flickr 30K test set for the embodiments of the disclosure;

[0026] Figure 4 The influence of vector dimension on cross-modal retrieval performance on MS-COCO 5K test set for the embodiments of the disclosure;

[0027] Figure 5 The image region context relationship attention visualization example for the embodiments of the disclosure; Figure 5 (A) in (A) is a visualization example A, Figure 5 (B) in (B) is a visualization example B;

[0028] Figure 6 Some retrieval examples of CMCL on MS-COCO dataset for I→T and T→I for the embodiments of the disclosure;Figure 6 (A) in the above (B) contains three groups of T→I retrieval examples. Figure 6 (A) in the above (B) contains three groups of T→I retrieval examples. DETAILED DESCRIPTION

[0029] The present disclosure will be further described below with reference to the drawings and examples.

[0030] It should be noted that the following detailed description is illustrative only and is intended to provide further description of the present disclosure. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs.

[0031] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit exemplary embodiments according to the present disclosure. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, there is a presence of a feature, step, operation, device, component, and / or combinations thereof.

[0032] Example 1

[0033] In an embodiment of the present disclosure, a context-guided multimodal association-based image-text retrieval method is provided, comprising:

[0034] Step 1: Obtain image and text data of different modalities, respectively extract fine-grained feature sequences of image and text data of different modalities; construct a context-guided multimodal association learning network framework, use fine-grained feature sequences of different modalities, capture complementary semantic relationships within modalities based on self-attention mechanism and gating mechanism, respectively obtain regionally spatially enhanced visual context perception representation of image modality and sequentially enhanced text context perception representation of text modality;

[0035] Step 2: Design global inter-modal association mining branch, local inter-modal association mining branch and local intra-modal association mining branch in the context-guided multimodal association learning network framework; in the global inter-modal association mining branch, cross-modal similarity learning is performed on the global level vectors of image and text; in the local inter-modal association mining branch, the cross-attention mechanism is used to fully capture the multimodal correlation between image regions and text words, and the similarity between image and text is learned at the local level; in the local intra-modal association mining branch, a graph model is constructed and graph convolution network is used to transmit and update information between nodes in the graph, and GRU network model is used to integrate and reason the relationship between different modalities;

[0036] Step 3: Introduce a cross-modal consistency term in the hybrid loss function and use a cross-modal alignment term to achieve effective alignment between different modes.

[0037] As one embodiment, the context-guided multimodal association learning network framework disclosed herein consists of three parts: 1) a context-guided mechanism, 2) multimodal association learning, and 3) hybrid loss function optimization.

[0038] The context-guided mechanism aims to perceive contextual information in images and text, and obtain context-aware visual and textual representations. Specifically, it utilizes the Stanford CoreNLP natural language processing toolkit to resolve semantic dependencies between words, facilitating the generation of more refined feature representations. This includes Context-Perceived Cells (CPCs), which leverage self-attention and gating mechanisms to fully utilize complementary semantic relationships within each modality and effectively capture contextual information within each modality. By suppressing informationless interactions between fine-grained features, it obtains visual context-aware representations of regions and textual context-aware representations of words in both image and text modalities.

[0039] The context-aware unit integrates a self-attention mechanism and a gating mechanism. The process of capturing intramodal contextual relationships based on the attention mechanism and the gating mechanism is as follows: the three input feature sets Q, K, and V of the self-attention mechanism are obtained by using a fully connected layer. The intramodal contextual information is mined by calculating the dot product similarity between Q and K. The gating mechanism adaptively adjusts the information flow within the modality and suppresses useless interaction relationships within the modality.

[0040] like Figure 2 As shown, the context-aware unit integrates a self-attention mechanism and a gating mechanism to adaptively acquire context-aware representations of fine-grained information within a modality.

[0041] Assume the input feature sequence of the context-aware unit is ,in Indicates the sequence length. This represents the feature dimension. Fully connected layers (FC) are used to obtain the three sets of input features for the self-attention mechanism, namely... (Query) (Key) (Value):

[0042] (1)

[0043] in, This is the weight matrix that needs to be trained. The self-attention mechanism calculates... and The dot product similarity between

[0044] (2)

[0045] where contains the original self-attention mechanism output of the intra-modal context information, but and may contain noise interference. In order to effectively capture the complementary semantic information of fine-grained features, we use a gating mechanism to adaptively adjust the information flow within the modal, and suppress the useless interaction within the modal.

[0046] First, perform the fusion operation for and :

[0047] (3)

[0048] where is the result of fusion, represents the element-wise product. Then, the gating mask matrix and are obtained by the full connection layer and sigmoid function respectively:

[0049] (4)

[0050] where represents the sigmoid function, are the weight matrix and bias term to be learned in the full connection layer respectively. Finally, the obtained gating mask matrix is used to control the information flow of and , and obtain:

[0051] (5)

[0052] The updated and are used to effectively mine the complementary semantic information within the modal, and equation (2) (4.2) is updated to:

[0053] (6)

[0054] where contains more meaningful context information within the modal, and ​​Therefore, the principle of the Context-Perceived Cell (CPC) can be described as follows:

[0055] (7)

[0056] As one embodiment, the process of obtaining the visual context-aware representation of the image modality with enhanced regional spatiality is as follows: extract salient regions of the image modality in the Faster R-CNN object detection model with a top-down attention mechanism, extract the features of the salient regions, map them to the common embedding space using a fully connected layer, obtain the position vectors of the salient regions, learn the absolute position features of the regions using the position vectors, integrate the features of each salient region and the position features corresponding to the salient regions, and then use the context-aware unit to capture the context information of the visual regions.

[0057] Specifically, for each image The Faster R-CNN object detection model, pre-trained on the Visual Genome dataset with bottom-up attention, was used to extract the first... We identified the most salient regions with the highest confidence scores and extracted features from these regions using ResNet-101, representing... ,in The dimensions of the features are represented. Then, fully connected layers are used to map them to... In the public embedded space of dimensions:

[0058] (8)

[0059] in, These are the weight matrix and bias term that need to be learned. It is an image New regional characteristics.

[0060] Unlike previous studies that used relative location features of regions, this paper proposes using absolute location features of regions to learn the spatial complementarity relationships between different regions from a global perspective. Specifically, firstly, assuming the image... The coordinates of the top-left and bottom-right corners are respectively and , The location features of each image region are represented as follows: , No. Location vector of each region Represented as: ,in, and They represent the first coordinates of the top-left and bottom-right corners of each region. Next, the absolute position features of regions are learned by using a fully connected layer and a sigmoid function :

[0061] (9)

[0062] where are the weight matrix and bias term to be learned, respectively. Thus, the new position feature of all regions in an image is represented as .

[0063] Finally, in order to fully exploit the complementary semantic information among different regions, we first integrate the feature of each region and its corresponding position feature, and then use the context perception unit to capture the context information of visual regions:

[0064] (10)

[0065] where denotes element-wise multiplication, is the spatial-enhanced visual context-perceived representation.

[0066] The pre-trained BERT (Bidirectional Encoder Representations from Transformers) model is used to process the text modality to obtain the context-related bidirectional feature representation. Specifically, for a sentence containing words, we first use the WordPiece tokenizer to tokenize the sentence, and then use BERT to extract the corresponding word features, denoted as . Similarly, we use a fully connected layer to map them to a -dimensional common embedding space:

[0067] (11)

[0068] where are the weight matrix and bias term to be learned, respectively, is the new region feature of the image .

[0069] Finally, the context perception unit is used to further integrate the context information of the word sequence in the text:

[0070] (12)

[0071] wherein, is a sequential-enhanced textual context-aware representation.

[0072] As an embodiment, in step two, a global inter-modal correlation mining branch, a local inter-modal correlation mining branch and a local intra-modal correlation mining branch are constructed in the context-guided multi-modal correlation learning network framework, and the specific implementation process includes:

[0073] Based on the context-aware representation of the image modal and the text modal, a global inter-modal correlation mining branch, a local inter-modal correlation mining branch and a local intra-modal correlation mining branch are constructed in the context-guided multi-modal correlation network learning framework. The global inter-modal correlation mining branch and the local inter-modal correlation mining branch respectively mine the correlation between different modalities from the global level and the local level, while the local intra-modal correlation mining branch learns the intra-modal correlation.

[0074] In order to capture more detailed complementary relationships between different modal features, a vector similarity function (VSF) is used in the three alignment modules. Suppose there are vectors and The vector similarity between them is defined as:

[0075] (13)

[0076] wherein, respectively represent element-wise square and norm, is a weight matrix that needs to be trained. In particular, the dimension of the similarity vector learned in the three alignment modules is set to .

[0077] 1) Global inter-modal correlation learning branch alignment module: learn a feature vector for each instance of different modalities, which reflects the inter-modal correlation between images and texts at the global level. For the visual context-aware representation and the text context-aware representation, the maximum pooling feature and the average pooling feature are obtained, the former highlights the importance of discriminative features, and the latter ensures the integrity of intra-modal information. Fuse them, and use a fully connected layer to learn the fused features.

[0078] Specifically, a global level vector is learned for each instance of different modalities, which reflects the inter-modal correlation between images and texts at the global level. For the visual context-aware representation and textual context-aware representations , firstly, get their MaxPooling features and Average Pooling features , the former highlights the importance of discriminative features, and the latter guarantees the integrity of intra-modal information, and fuse them respectively:

[0079] (14)

[0080] Then, further utilize the fully connected layer to learn the fused features, and get the final global-level vectors of image and text respectively

[0081] (15)

[0082] where, and are the weight matrix and bias term of the fully connected layer respectively.

[0083] Finally, learn the inter-modal correlation between image and text at the global level by calculating the similarity between them:

[0084] (16)

[0085] 2) Inter-modal correlation learning alignment module at the local level: Since fine-grained fragments can provide important complementary semantic information, this branch fully captures the multi-modal correlation between image regions and words through cross-attention mechanism, so as to learn the similarity between image and text at the local level.

[0086] For the input visual context-aware representation and textual context-aware representation, firstly, calculate the region-word similarity matrix, then use cross-attention in two directions of visual context-aware representation to textual context-aware representation and textual context-aware representation to visual context-aware representation, to obtain the complementary relationship between regions and words, in the direction of visual context-aware representation to textual context-aware representation, learn a fusion vector of all words in the text for each region, then calculate the similarity between each region and the corresponding word fusion vector, to get the similarity of image and text in the direction of visual context-aware representation to textual context-aware representation. At the same time, obtain the similarity of image and text in the direction of textual context-aware representation to visual context-aware representation, add the similarities obtained from the two directions to learn the inter-modal correlation at the local level. ​​

[0087] Specifically, in order to uncover richer local complementary information between multimodal data, a cross-attention mechanism is used to fully capture fine-grained alignment between image regions and words.

[0088] For the visual context-aware representation of the input and text context-aware representation First, calculate the region-word similarity matrix. ,in Indicates the region With words The cosine similarity between them. Then, in and Cross-attention is applied in both directions to fully explore the complementary relationships between regions and words. Direction, for each region Learn the fused vector of all words in a text ,in These are the cross-attention weights, calculated as follows:

[0089] (17)

[0090] in, It is The elements in the matrix are normalized along the column dimensions. Then, each region is calculated. fused with corresponding word vectors The similarity between them is calculated, and the average of all the obtained similarities is used as the average value. Image in direction and text Similarity:

[0091] (18)

[0092] Similarly, by analyzing the word-region similarity matrix... By performing the above processing, we can obtain Image in direction and text Similarity:

[0093] (19)

[0094] Finally, the similarity vectors obtained from the two directions are added together to represent the image. and text Alignment at the target level to learn intermodal relationships at the local level:

[0095] (20)

[0096] 3) Local-level intra-modal association learning alignment module: In local-level intra-modal association learning, a visual graph is constructed based on visual context-aware representation, and a text graph is constructed based on text context-aware representation. The grammatical dependency matrix between words in the text is obtained, and the affinity matrix between nodes in the text graph is calculated. A graph convolutional network is used to transmit and update information between nodes in the graph. Finally, a GRU network model is used to integrate and infer the relationships between different modalities.

[0097] Specifically, the first two branch modules integrate complementary semantic information within the modality from the global and local levels, respectively, but cannot learn the correspondence between different objects and their relationships and attributes in the modality. Therefore, they have two limitations: (1) due to the large number of object correspondences, it is difficult to learn the correspondence between relationships and attributes; (2) without the guidance of descriptive relationships and attributes, objects are prone to being assigned to the wrong category. On the one hand, for the image modality, the location features of the region well describe the spatial attributes of different objects in the image, and the graph model can be used to effectively capture the spatial relationship information between regions. First, visual graphs and text graphs are constructed for the image and text respectively, and then the graph convolutional network is used to transmit and update the information between nodes in the graph, and the GRU network model is used to integrate and reason about the relationship between different modalities.

[0098] Visual graph construction: Context-aware representation Each image is represented as a fully connected undirected graph. ,in It is the set of all nodes in the graph. It is the set of all edges in the graph. Specifically, each node in the graph corresponds one-to-one with a region of the image. Indicates the first The features of each node, and the fact that all nodes are connected by edges, mean that information can be transferred between all image regions. To accurately describe the relationships between different image regions, we propose calculating their affinity as weights on the edges in the graph:

[0099] (twenty one)

[0100] in, and These are two types of embedding features. It is the mapping matrix that needs to be trained. As the first in the figure The node and the first The weights of the edges between nodes. Therefore, the edge weight matrix of the visual graph can be obtained. .

[0101] Text graph construction: As shown in Figure 1 , given a sentence "A man leads a white and black puppy.", where "man" and "puppy" are the target entities (noun), "a", "white" and "black" are the attributes (adjective / adverb) of the entity "puppy", and "leads" is an action (verb) of "man". It can be found that there are interpretable syntactic dependency relationships in the text, so it is not suitable to construct a fully connected graph for it, because it may increase false associations, which will interfere with the relationship reasoning between subsequent nodes. Therefore, the syntactic dependency relationships in the text are parsed using Stanford CoreNLP to construct a text graph , where the features of the nodes in the graph use text context-aware representations , and the syntactic dependency matrix between words in the text is obtained by Stanford CoreNLP , where if there is a syntactic dependency relationship between word and word , then , otherwise . Similar to the visual graph, the affinity matrix between nodes in the text graph is calculated to emphasize the intrinsic relationship between words:

[0102] (22)

[0103] where are also two kinds of embedding features that need to be learned using mapping matrices, and the affinity matrix is fused with the syntactic dependency matrix as the edge weight matrix of the text graph:

[0104] (23)

[0105] where denotes element-wise multiplication, is the edge weight matrix of the text graph.

[0106] Graph Convolutional Networks (GCN) is a model applied to graphs, which aggregates the information of neighboring nodes through the edge weight matrix when updating the node features, so as to capture the potential relationship between different nodes. In order to fully exploit the relationship complementarity between fine-grained features in different modalities, Residual Graph Convolutional Networks (RGCN) is used to infer the association relationship between nodes in the graph.

[0107] Specifically, given a graph with N nodes , define its node feature matrix as , and the edge weight matrix as , the process of updating the node features using RGCN is as follows:

[0108] (24)

[0109] where is the parameter matrix to be trained in the graph convolution network, is the residual structure matrix, and the updated node feature matrix contains the reasoning relationship between nodes.

[0110] Therefore, the node relationship reasoning process on the visual graph and the text graph is as follows:

[0111] (25)

[0112] where and are the parameter matrix and residual structure matrix in the visual graph, and are the parameter matrix and residual structure matrix in the text. By updating the node features using the graph, we obtain the region features and word features enhanced by inter-modal neighbor relationships.

[0113] To further select the discriminative fine-grained interaction relationship information in each modality and filter out the redundant part, first combine the global level vector learned in branch 1 (i.e. ) of one modality with the neighbor relationship enhanced features of the other modality, and then input them into the GRU network model to simultaneously fuse the inter-modal and intra-modal relationship complementarity. Specifically, the features in are input into the GRU in turn:

[0114] (26)

[0115] where is the feature of the last layer of the GRU hidden state, which is taken as the global relationship vector of the image and the global relationship vector of the text, respectively.

[0116] Finally, the similarity between and is calculated to learn the intra-modal correlation relationship:

[0117] (27)

[0118] Further, after fully exploiting the complementary semantic information between different modalities, the cross-modal similarities learned by different branches will be accurately optimized.

[0119] On the one hand, since the cross-modal similarity essentially depends on the shared attributes between different modalities, these attributes should not change in different embedding spaces, so the cross-modal similarities learned by different modules should not differ too much, therefore we innovatively propose a cross-modal consistency term to constrain the difference of cross-modal similarities learned by different modules. On the other hand, we want the cross-modal similarity between matched images and texts to be higher than that between unmatched images and texts, so we use a cross-modal alignment term to achieve effective alignment between different modalities. By integrating the two terms, we obtain a hybrid loss function to achieve accurate optimization of cross-modal similarity.

[0120] Specifically, first, we use a multi-layer perceptron (MLP) with three sets of shared parameters to map the three similarity vectors obtained by the multi-branch alignment module (i.e. , , ) into scalars (i.e. , , ) respectively. The multi-layer perceptron consists of two fully connected neural networks, and after the last layer, it outputs the learned cross-modal similarity through a softmax activation function. Let and represent the input and output respectively, and the learning process is as follows:

[0121] (28)

[0122] where and are the weight matrix and bias term to be learned respectively.

[0123] Cross-modal consistency term

[0124] Since the calculation of cross-modal similarity depends on connecting different modalities based on shared semantic information, the differences between the embedding spaces learned by our three branches should be as small as possible. Therefore, we propose a cross-modal consistency term to limit the differences between different embedding spaces. Specifically, the difference between image-text similarity and is defined as:

[0125] (29)

[0126] Therefore, the images we learned earlier and text Three types of cross-modal similarity (i.e.: , , The differences are as follows:

[0127] (30)

[0128] Therefore, for each batch of data during the training process The cross-modal consistency term is defined as:

[0129] (31)

[0130] Cross-modal alignment terms

[0131] To achieve semantic alignment between different modalities, the image is first... and text Cross-modal similarity between Defined as the average of the three similarities mentioned above:

[0132] (32)

[0133] Then, a hinged bidirectional triplet loss function is used to optimize cross-modal similarity, and to improve computational efficiency, in each batch during training, i.e. A hard negative sample mining strategy is employed. The cross-modal alignment loss function is defined as follows:

[0134] (33)

[0135] in, It refers to the batch size. Indicates the first The image and the first Similarity between texts , and It is the negative example in this batch that is most similar to the current query term. It is a predefined interval coefficient.

[0136] Finally, the cross-modal alignment loss function and the semantic consistency loss function are combined as the overall objective function:

[0137] (34)

[0138] in, is a balancing factor.

[0139] The present disclosure performs a series of experiments on two benchmark datasets, Flickr 30K and MS-COCO, and compares the performance with a variety of recently proposed advanced methods, verifying the superiority of CMCL. In addition, detailed parameter analysis of the hyperparameters involved in CMCL is performed, and the effectiveness of each module in CMCL is proved through ablation experiments. Finally, some attention visualization results and retrieval example results are shown, such as Figure 3 and Figure 4 .

[0140] As shown in Figure 5 , two original images with a large number of visual objects and complex relationships between A and B are shown, and the corresponding regional attention visualizations of the images are also shown. In addition, the sentences corresponding to each image are given. From the attention visualization of the image regional context relationship, it can be found that the image representation generated by CMCL well captures the key objects in the image and their context relationships. For example, the context relationships between the object "Skiers" and the related "on their skis", and "ride on the slope" in the first group of examples; the context relationships between the object "man" and the related "in red shirt and red hat", and "on a motorcycle" in the second group of examples are well reflected.

[0141] In addition, Figure 6 some retrieval examples of CMCL on the MS-COCO dataset are shown. Figure 6 (A) in lists three groups of I→T retrieval examples. For each query image, the top five (Rank1-Rank5) texts in the CMCL retrieval results are shown, Figure 6 (B) in contains three groups of T→I retrieval examples. For each text query, the top five (Rank1-Rank5) images in the CMCL retrieval results are shown.

[0142] Embodiment 2

[0143] In an embodiment of the present disclosure, an image-text retrieval system based on context-guided multi-modal association is provided, comprising:

[0144] The modal data representation module acquires image and text data of different modalities, and extracts fine-grained feature sequences of the image and text data of different modalities respectively; a context-guided multi-modal correlation network framework is constructed, and fine-grained feature sequences of different modalities are used to capture complementary semantic relationships within the modalities based on a self-attention mechanism and a gating mechanism, so as to acquire regionally spatially enhanced visual context perception representation of the image modality and sequentially enhanced text context perception representation of the text modality respectively;

[0145] The three-branch multi-modal correlation relationship mining module is used for designing a global inter-modal correlation mining branch, a local inter-modal correlation mining branch and a local intra-modal correlation mining branch in the context-guided multi-modal correlation learning network framework; in the global inter-modal correlation mining branch, cross-modal similarity learning is performed on global level vectors of the image and the text; in the local inter-modal correlation branch, a cross-attention mechanism is used to fully capture multi-modal correlation between image regions and text words, and similarity between the image and the text is learned at a local level; in the local intra-modal correlation mining branch, a graph model is constructed, and a graph convolution network is used to transmit and update information between nodes in the graph, and a GRU network model is used to integrate and reason the relationship between different modalities;

[0146] The alignment module is used for proposing a cross-modal consistency term in a hybrid loss function, and using a cross-modal alignment term to realize effective alignment between different modalities.

[0147] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing device to cause a series of operation steps to be performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed by the computer or other programmable device provide a process for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0149] The specific embodiments of the present disclosure are described above with reference to the accompanying drawings, but are not intended to limit the protection scope of the present disclosure, and those skilled in the art should understand that various modifications or changes made on the basis of the technical solutions of the present disclosure without creative labor are still within the protection scope of the present disclosure.

Claims

1. An image-text retrieval method based on context-guided multimodal association, characterized in that, include: Acquire image and text data of different modalities, and extract fine-grained feature sequences from the image and text data of different modalities respectively; A context-guided multimodal association learning network framework is constructed. By utilizing fine-grained feature sequences from different modalities, complementary semantic relationships within the modalities are captured based on self-attention and gating mechanisms. This results in the acquisition of regional spatially enhanced visual context-aware representations for image modalities and sequentially enhanced text context-aware representations for text modalities. The context-guided multimodal association learning network framework includes a context-guided mechanism, multimodal association learning, and hybrid loss function optimization; The context guidance mechanism includes a context-aware unit, which integrates a self-attention mechanism and a gating mechanism; In the context-guided multimodal association learning network framework, a global intermodal association mining branch, a local intermodal association mining branch, and a local intramodal association mining branch are designed. In the global intermodal association mining branch, cross-modal similarity learning is performed on the global-level feature vectors of images and text. In the local modality association mining branch, the cross-attention mechanism is used to fully capture the multimodal correlation between image regions and text words, and the similarity between images and text is learned at the local level. In the local modality association mining branch, a graph model is constructed and a graph convolutional network is used to pass and update information between nodes in the graph. The GRU network model is used to integrate and infer the relationship between different modalities. The cross-modal similarity learned in the global intermodal association mining branch, the local intermodal association mining branch, and the local intramodal association mining branch is optimized by using a multilayer perceptron with three sets of shared parameters; In the hybrid loss function, a cross-modal consistency term and a cross-modal alignment term are proposed to achieve effective alignment between different modes.

2. The image-text retrieval method based on context-guided multimodal association as described in claim 1, characterized in that, The process of capturing the contextual relationships within each modality based on self-attention and gating mechanisms is as follows: three sets of input features Q, K, and V for the self-attention mechanism are obtained by using fully connected layers. The contextual information within the modality is mined by calculating the dot product similarity between Q and K. The gating mechanism is used to adaptively adjust the information flow within the modality and suppress useless interaction relationships within the modality.

3. The image-text retrieval method based on context-guided multimodal association as described in claim 1, characterized in that, The process of obtaining a visual context-aware representation of an image modality with enhanced regional spatiality is as follows: extract salient regions of the image modality in a target detection model with a top-down attention mechanism, extract the features of the salient regions, map them to a common embedding space using a fully connected layer, obtain the position vectors of the salient regions, learn the absolute position features of the regions using the position vectors, integrate the features of each salient region and the position features corresponding to the salient regions, and then use a context-aware unit to capture the contextual information of the visual regions.

4. The image-text retrieval method based on context-guided multimodal association as described in claim 1, characterized in that, The process of obtaining the sequentially enhanced text context-aware representation of the text modality is as follows: First, the text is segmented into words, and the corresponding word features are extracted using BERT. Then, the features are mapped to the common embedding space using a fully connected layer. Finally, the context information of the word sequence in the text is further integrated using a context-aware unit.

5. The image-text retrieval method based on context-guided multimodal association as described in claim 1, characterized in that, The global intermodal correlation mining branch and the local intermodal correlation mining branch respectively mine the correlations between different modalities at the global and local levels, while the local intramodal correlation mining branch learns the intramodal correlation relationships.

6. The image-text retrieval method based on context-guided multimodal association as described in claim 1, characterized in that, The three similarity vectors obtained by the multi-branch alignment module are mapped to scalars by a multi-layer perceptron with three sets of shared parameters. The multi-layer perceptron consists of two fully connected neural networks, and the learned cross-modal similarity is output through the softmax activation function after the last layer.

7. The image-text retrieval method based on context-guided multimodal association as described in claim 6, characterized in that, A cross-modal consistency term is proposed to constrain the differences between different embedding spaces, defining the differences between image and text similarity.

8. The image-text retrieval method based on context-guided multimodal association as described in claim 1, characterized in that, To achieve semantic alignment between different modalities, the cross-modal similarity between images and text is first defined as the average of three similarities. A hinged bidirectional triplet loss function is used to optimize the cross-modal similarity, and a hard negative sample mining strategy is adopted for each batch during the training process. The cross-modal alignment loss function and the semantic consistency loss function are combined as the overall objective function.

9. An image-text retrieval system based on context-guided multimodal association, characterized in that, include: The modal data representation module acquires image and text data of different modalities and extracts fine-grained feature sequences from the image and text data of different modalities respectively; A context-guided multimodal association network framework is constructed. By utilizing fine-grained feature sequences from different modalities, complementary semantic relationships within modalities are captured based on self-attention and gating mechanisms. This results in the acquisition of regional spatially enhanced visual context-aware representations for image modalities and sequentially enhanced text context-aware representations for text modalities. The context-guided multimodal association learning network framework includes a context-guided mechanism, multimodal association learning, and hybrid loss function optimization; The context guidance mechanism includes a context-aware unit, which integrates a self-attention mechanism and a gating mechanism; The three-branch multimodal association mining module is used to design a global intermodal association mining branch, a local intermodal association mining branch, and a local intramodal association mining branch in the context-guided multimodal association learning network framework; in the global intermodal association mining branch, cross-modal similarity learning is performed on the global level vectors of images and text; In the local modality association mining branch, the cross-attention mechanism is used to fully capture the multimodal correlation between image regions and text words, and the similarity between images and text is learned at the local level. In the local modality association mining branch, a graph model is constructed and a graph convolutional network is used to pass and update information between nodes in the graph. The GRU network model is used to integrate and infer the relationship between different modalities. The cross-modal similarity learned in the global intermodal association mining branch, the local intermodal association mining branch, and the local intramodal association mining branch is optimized by using a multilayer perceptron with three sets of shared parameters; The alignment module is used to propose cross-modal consistency terms in the hybrid loss function and to achieve effective alignment between different modes using cross-modal alignment terms.

Images