An image-text matching method based on a graph neural network

Abstract

The application discloses an image-text matching method based on a graph neural network, comprising the following steps: obtaining image data and text data, and obtaining image representation and text representation respectively through attention calculation, and mapping the image representation and the text representation to a shared embedding space; in the shared embedding space, aligning the region features in the image representation with the word features in the text representation; and inputting into a graph neural network for similarity reasoning to generate a similarity matrix; according to the similarity matrix, using a quadratic re-ranking algorithm for image-to-text retrieval and text-to-image retrieval, and updating network parameters through dynamic triplet loss back propagation; the application uses the quadratic re-ranking algorithm, so that the model can fully utilize the information of bidirectional retrieval in the retrieval stage, greatly improves the performance of text-to-image detection, and does not need to increase additional calculation in model training.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer image-text matching, and more particularly to an image-text matching method based on a graph neural network. BACKGROUND

[0002] Image-text matching refers to finding relevant text or images from a dataset by inputting an image or text. Therefore, image-text matching usually includes two sub-tasks: image-to-text matching and text-to-image matching. Studying image-text matching technology is helpful for many real-world scenarios, allowing people to better understand and utilize image information and improve the efficiency and convenience of various application scenarios.

[0003] Currently, the loss function of the image-text matching algorithm model mainly uses a triplet loss, which trains the model by maximizing the similarity distance between positive samples and minimizing the similarity distance between negative samples. Although this loss function can learn better feature representation and is highly adaptable to data distribution, it treats positive samples and negative samples as equally important in calculation, but in actual situations, the weights of positive and negative samples should be related to the situation of the dataset, and using the same weight may deviate from the real data. Secondly, since the commonly used dataset uses one image corresponding to five texts, the model has different performances for learning image-to-text matching and text-to-image matching.

[0004] Therefore, how to balance the two performances is a problem that those skilled in the art need to solve. SUMMARY

[0005] Therefore, the present application provides an image-text matching method based on a graph neural network, which realizes balanced application of bidirectional retrieval capability.

[0006] To achieve the above purpose, the present application adopts the following technical solutions:

[0007] An image-text matching method based on a graph neural network includes the following steps:

[0008] Obtain image data and text data, and obtain image representation and text representation respectively through attention calculation, and map the image representation and the text representation to a shared embedding space;

[0009] In the shared embedding space, align the region features in the image representation with the word features in the text representation, and calculate the similarity to generate a similarity matrix;

[0010] According to the similarity matrix, perform image-to-text retrieval and text-to-image retrieval using a quadratic re-ranking algorithm, and update the network parameters through dynamic triplet loss backpropagation.

[0011] Further, the step of obtaining the image representation by attention is:

[0012] performing attention calculation on the image data from bottom to top to obtain attention coefficients of each image region;

[0013] extracting a key image region according to the attention coefficients to obtain a region element;

[0014] inputting the region element into a classifier to identify image features of each region element to obtain the image representation of the image data.

[0015] Further, the step of obtaining the text representation by attention is:

[0016] unifying the dimension of the word embedding vector of the text data and the dimension of the region embedding vector of the image data by dimension conversion;

[0017] inputting the word embedding vector into a recurrent network to extract forward context word features and reverse context word features, and combining the forward context word features and the reverse context word features to obtain the text representation.

[0018] Further, the step of calculating the similarity includes: using a graph neural network to perform global alignment and local alignment respectively to calculate the global similarity and the local similarity of the image and the text, and fusing the two to obtain the total similarity.

[0019] Further, the step of calculating the global similarity includes:

[0020] constructing a multi-modal knowledge graph to train a neural network model, and reasoning the region features and the word features respectively by the trained neural network model to obtain a graph representation;

[0021] enhancing the global embedding of the input image and the global embedding of the input text by the graph representation to generate a multi-modal knowledge enhanced embedding;

[0022] calculating the global similarity of the image enhanced embedding and the text enhanced embedding by cosine similarity.

[0023] Further, the step of constructing a multi-modal knowledge graph to train a neural network model includes:

[0024] obtaining training images {I1,...,I N} and extracting triplets {I i , O i , T i} from the training images, wherein I i is an original image, O i is a list of image objects appearing in I i , and Ti is a text description of the image I i ;

[0025] A set of relations between image objects and text descriptions is constructed according to the triplets, and a co-occurrence matrix is calculated according to the co-occurrence times; a text path similarity matrix is calculated according to the text descriptions; an object path similarity matrix is calculated according to the object list;

[0026] The neural network combines the text path similarity matrix and the object path similarity matrix to perform reasoning, and the learnable matrix in the neural network is updated according to the reasoning result until convergence.

[0027] Further, the calculation step of the local similarity includes:

[0028] Image features in each region element and word features in each word element are obtained, and the attention weight of each region element is calculated;

[0029] The word visual feature is calculated according to the attention weight of the region element, and the similarity between the word visual feature and the word feature is calculated to obtain the participation similarity of each word feature;

[0030] The participation similarity of each word feature is taken as a node, the edges between nodes are calculated, a similarity graph is constructed, and each node and each edge are reasoned to obtain the local similarity.

[0031] Further, the reasoning of each node and each edge in the similarity graph includes:

[0032] The edge between any two nodes is calculated as:

[0033]

[0034] Where S p and S q are two arbitrary nodes, W in and W out are linear transformations of incoming and outgoing nodes, respectively;

[0035] The reasoning of the similarity graph is performed by updating the nodes and edges in the following way:

[0036]

[0037]

[0038] Where, and are learnable parameters.

[0039] Further, the calculation function of the dynamic triplet loss is:

[0040] L(I, T) = max(a*S(I, T) - (1 - a)*S(I, T') + m, 0) + max(b*S(T, I) - (1 - b)*S(T, I') + m, 0)

[0041] wherein a and b are learnable parameters, S(·,·) represents a similarity calculation function, I and T in the first parameter of S(·,·) represent the query image and the query text respectively, T and T' in the second parameter of S(·,·) represent the positive sample text and the negative sample text of the query image, and I and I' represent the positive sample image and the negative sample image of the query text.

[0042] Further, the step of searching by using the secondary reordering method comprises:

[0043] inputting an image, performing a first round of searching on the input image according to similarity to obtain K first similar texts most similar to the input image, obtaining M similar images most similar to each first similar text, embedding a position index for the similar text according to the position of the similar image to obtain an image position set corresponding to the input image, and outputting a first similar text sequence after sorting the image position set;

[0044] selecting a second similar text from the similar text sequence to perform a second round of searching, and outputting a second similar text sequence.

[0045] According to the technical solution described above, compared with the prior art, the present disclosure provides an image-text matching method based on a graph neural network, which has the following beneficial effects:

[0046] 1. The secondary reordering algorithm used in the present disclosure enables the model to fully utilize the information of bidirectional searching in the searching stage, greatly improves the performance of text-to-image detection, and does not need to increase additional calculation in model training.

[0047] 2. The present disclosure adds learnable parameters to adaptively learn the distribution of data, better adapt to different data sets, and automatically learn a better similarity measurement method, thereby increasing the learning ability of the model.

[0048] 3. The use of the graph neural network enables global alignment and local alignment to be performed respectively, fully utilizes all features, and makes the matching ability of the model stronger. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0050] Figure 1 A schematic diagram of an image-text matching method based on a graph neural network is provided. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0052] As Figure 1 The embodiments of the present application disclose an image-text matching method based on a graph neural network, comprising the following steps:

[0053] S1: Obtain image data and text data, and obtain image representation and text representation respectively through attention calculation, and map the image representation and the text representation to a shared embedding space. The image representation and the text representation are respectively composed of multiple image features and multiple word features;

[0054] S2: In the shared subspace, align the elements, measure the similarity between the image and the text by using the graph neural network, calculate the similarity of the image features and the text features, and generate a similarity matrix.

[0055] S3. According to the similarity matrix, use a quadratic re-ranking algorithm for image-to-text retrieval and text-to-image retrieval, and update the network parameters by dynamic triplet loss backpropagation.

[0056] In order to further implement the above technical solutions, in S1, the image representation refers to extracting key image regions using a bottom-up attention mechanism FasterR-CNN method. In order to let the model identify more details, attribute classification and instance classification (including sky, grassland, fur texture, etc.) are used to train the model, the output of the last convolutional layer is averaged and pooled, and the dimension is transformed through a fully connected layer. The function of the fully connected layer is:

[0057] v i =W v f i +b v

[0058] Finally, a picture is represented as: v={v1,...,v k}, where v is the feature of each region, and k is the number of regions.

[0059] In order to further implement the above technical solution, in S1, the text representation refers to converting the text into features with the same dimension as the image in order to compare the image and the text using the same scale, and the combined semantic information and the influence of the context also need to be considered. Each word is converted into a 300-dimensional word embedding, and then input into a bidirectional GRU (recurrent network) and output the last hidden layer h:

[0060]

[0061]

[0062] Finally, the word features combined with the bidirectional context before and after are output:

[0063]

[0064] In order to further implement the above technical solution, in S2, the similarity calculation uses a graph neural network to perform global alignment and local alignment respectively, and calculates the global similarity and local similarity of the image and the text. The total similarity can be obtained by fusing the two.

[0065] In order to further implement the above technical solution, in S2, the global similarity adds multi-modal knowledge to construct a multi-modal knowledge graph.

[0066] Entity selection VisualGenome and MSCOCO, Flickr30k training set N images (N is 47210) form an image set {I1,..., In} N}, and N triplets {I i , O i , T i} can be obtained according to the image set, wherein I i is the original image, O i is the image object list appearing in I i , and T i is the text description of the image I i .

[0067] Then, some meaningless words such as quantifiers and definite articles are ignored in the text set {T1,..., T N}, and nt most frequently appearing words are selected to form the text entity set ni most frequently appearing objects are selected from the set {O1,..., O N} as the image object entity set

[0068] The co-occurrence number of the relationship set is calculated according to the N triplets {I i , O i , Ti} is counted. The co-occurrence matrix is denoted as The path similarity matrix of text words is denoted as The path similarity matrix of image objects is denoted as The two path similarity matrices can be represented as:

[0069]

[0070] where s p (·,·) is the path similarity, which is calculated by the "path similarity" in the Natural Language Toolkit (NLTK). The path similarity calculation formula is s p (·,·) = 1 / (1+d(·,·)), where d(·,·) represents the shortest path distance between two words in the is-a (hyponym) taxonomy, which returns a score indicating the similarity of the two words. The score is in the range of 0 to 1, where 1 represents the maximum similarity and 0 represents the minimum similarity. Path similarity can help distinguish other entities in the semantic space.

[0071] The representation of an entity, for each text entity T i is embedded into a vector g i ∈R300 using the co-occurrence matrix by GloVe. i For each image object entity O i , the original image set containing O is denoted as i , where k i represents the total number of the above original images. The embedding of each O is denoted as the average of the features of all I i 's previously extracted R key regions, i.e.:

[0072]

[0073] The average pooling of image features belonging to the same category can represent the average semantics of each object category. Finally, the representation of text and image entities is:

[0074]

[0075] Different modal entities of GCN will have different features, thus forming different types of relationships. In order to fully utilize the multi-modal knowledge implied in the multi-modal knowledge graph, GCN is used for reasoning. GCN can generate semantic features for each entity to distinguish other entities in the semantic space. Specifically, first, the relationship matrix A, A i and A t is normalized as:

[0076] ​

[0077] The entity embeddings G and B in the multi-modal knowledge graph are linearly projected to a D-dimensional space through a fully connected layer, to obtain and The GCN respectively reasons the text entity and the image entity:

[0078]

[0079]

[0080] wherein, l m is the total number of layers of the GCN, and is the adjacency matrix of the GCN, which is pre-calculated by the formula and then fixed, and are learnable matrices, and are learnable bias matrices, and σ(·) is a LeakyReLU activation function, which can be represented as:

[0081]

[0082] wherein, α is a constant greater than 0, and here the value is 0.1.

[0083] Embedding enhancement, the global embedding of the input image and the global embedding of the input text are enhanced by the entity representation in the multi-modal knowledge graph to generate multi-modal knowledge enhanced embeddings for similarity calculation. Specifically, a multi-head attention mechanism is used to encode the global embedding of the input image and and the global embedding of the input text as follows:

[0084] MultiHead(X,Y)=Concat(h1,...,h H )+X

[0085] The global embedding can be completed by average pooling, that is,

[0086]

[0087]

[0088] t i and v i ​represents one dimension in the text local embedding and the image local embedding respectively; wherein the local embedding of the image refers to the matrix embedded after the image passes through the Faster R-CNN and ResNet network; and the local embedding of the text refers to the matrix embedded after passing through the embedding and bidirectional GRU.

[0089] wherein X is or Y is or Concat(·) represents a connection operation on a feature dimension alone, H represents the number of attention heads, h i The scaled dot-product attention mechanism Att(·) is used for calculation:

[0090]

[0091]

[0092] wherein the Softmax function acts on each row of the matrix, d k is the number of channels of Q and K, and are learnable matrices. Then the attention results from different heads are combined through two fully connected layers. Based on the above process, the multi-modal knowledge enhanced embedding can be represented as follows:

[0093]

[0094] wherein FFN(·) represents two layers of fully connected layers with ReLU activation functions in between. The final global embedding can be represented as:

[0095]

[0096] wherein λ c is a hyperparameter of the connection weight of the multi-modal knowledge enhanced embedding.

[0097] Finally, the global similarity s glo between the image and the text can be obtained by the cosine similarity:

[0098] s glo =cos(v glo , t glo ).

[0099] In order to further implement the above technical solutions, in S2, the local similarity adopts text-to-visual attention to focus on the relationship between each region and each word. The attention weight of each region is calculated by the following formula:

[0100]

[0101]

[0102] where denotes the temperature parameter of the Softmax function, c ij is the cosine similarity between region features v i and word features t j , is the normalized similarity matrix of c ij , and [·] + denotes taking 0 or the maximum value in the parameter. Then, the visual feature of the j-th word participation a ij is generated according to a

[0103]

[0104] The similarity between a and t j can then be represented as:

[0105]

[0106] To achieve more comprehensive similarity inference, this paper constructs a similarity graph to propagate similarity messages between all possible alignments at the local level. Specifically, the similarity of all word participations is represented as a graph node, i.e. The edge between two nodes s p , s q ∈ N in the graph is calculated as:

[0107]

[0108] where W in and W out are linear transformations of incoming and outgoing nodes, respectively. With the constructed nodes and edges of the graph, the similarity graph inference is performed by updating the nodes and edges as follows:

[0109]

[0110]

[0111] where, at step n = 0, and are taken from N, and are learnable parameters, and after the current step of graph inference, the node is replaced by After the network is iterated for N steps, it is sent to a fully connected layer to obtain the final local similarity s loc : ​

[0112]

[0113] Global similarity s glo And local similarity s loc The total similarity s is obtained:

[0114] s = s glo + s loc .

[0115] In order to further implement the above technical solution, in S3, the dynamic triple loss function refers to the model loss function, which can effectively reduce the data dimension, improve the model generalization ability, and has good interpretability, can intuitively understand the similarity relationship between samples, and can solve the problem of small intra-class difference and large inter-class difference. The samples from the same class are compressed together, and the samples from different classes are dispersed. The specific formula is as follows:

[0116] L(I,T)=max(α*S(I,T)-(1-α)*S(I,T′)+m,0)+max(β*S(T,I)-(1-β)*S(T,I′)+m,0)

[0117] In order to further implement the above technical solution, in S3, in the dynamic triple loss function, alpha and beta are learnable parameters, S(·,·) represents a similarity calculation function, I and T in the first parameter of S(·,·) represent query image and query text respectively, T and T' in the second parameter represent positive sample text and negative sample text of the query image, and I and I' represent positive sample image and negative sample image of the query text.

[0118] In order to further implement the above technical solution, in S3, the secondary reordering algorithm includes image-to-text reordering and text-to-image reordering.

[0119] In order to further implement the above technical solution, in S3, in the image-to-text reordering, for a query image I, the initial K most similar texts can be obtained according to the similarity matrix between the image and the text, and these candidate text sequences are denoted as R I2T (I,K), which can be represented as

[0120] R I2T (I,K)={T1,..., Tk} j ,..., Tk} K

[0121] The value of K is the number of test set texts N, that is, the similarity between image I and all texts is sorted.

[0122] For each candidate text T j ​In other words, we can obtain the M most similar images using the image-text similarity matrix, and this set is denoted as R. T2I (T j M) can be represented as:

[0123] R T2I (T j M) = {I1, ..., I s , ..., I M}

[0124] Where M is the number of images in the test set. To fuse bidirectional nearest neighbors R... T2I and R T2I Further, for each candidate text T j Introduce a position index p(T) j ):

[0125] p(T j ) = s, if I s =I,I s ∈R T2I (T j M)

[0126] Set the position index p(T) j Replace R I2T The corresponding part in (I, K) can be used to obtain the location set p(I, K) of image I:

[0127] p(I, K) = {p(T1), ..., p(T)} j ), ..., p(T) K )}

[0128] The set p(I, K) can be viewed as a set of initial sequences R. I2T The reordering of (I, K) only requires sorting the set p(I, K) to obtain the K most similar text sequences of the reordered image I:

[0129] R′ I2T =ranking(p(I,K))

[0130] Among them, R′ I2T This represents the retrieval list of image I after reordering, which is the result of the first round of reordering.

[0131] The second round of reordering differs slightly from the first round; it requires starting from R′. I2T Select the top K texts as the new candidate sequence R′ I2T (I, K):

[0132] R′ I2T (I, K) = {T′1, ..., T′} j..., T' K}

[0133] where K does not need to take N, thus reducing unnecessary computation. The remaining steps are the same as the original reordering, as follows:

[0134] R T2I (T j , M) = {I'1,..., I' s ,..., I' M}

[0135] p(T' j ) = s, if I s = I, I s ∈ R T2I (T j , M)

[0136] p'(I, K) = {p(T'1),..., p(T' j ),..., p(T' K )}

[0137] R" I2T = ranking(P'(I, K))

[0138] R" I2T , i.e., the retrieval list of image I after the second reordering.

[0139] To further implement the above technical solution, in S3, the text-to-image reordering no longer needs to calculate the single-modal neighbor set of query text T according to the text-to-text similarity matrix, and can be calculated according to the image-to-text second reordering algorithm described above. Finally, the retrieval list R" of text T after the second reordering can be obtained. I2T .

[0140] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0141] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A graph neural network-based image-text matching method, characterized in that, The method comprises the following steps: obtaining image data and text data to obtain image representation and text representation respectively, and mapping the image representation and the text representation to a shared embedding space; aligning region features in the image representation with word features in the text representation in the shared embedding space; and inputting into a graph neural network for similarity inference to generate a similarity matrix; the similarity inference comprises: using the graph neural network to perform global alignment and local alignment respectively, to calculate global similarity and local similarity of the image and the text, and to obtain total similarity by fusing the two; the calculation steps of the global similarity comprise: constructing a multi-modal knowledge graph training neural network model, and reasoning the region features and the word features respectively by using the trained neural network model to obtain graph representation; enhancing the global embedding of the input image and the global embedding of the input text through the graph representation to generate multi-modal knowledge enhanced embedding; calculating the global similarity of the image enhanced embedding and the text enhanced embedding by using cosine similarity; according to the similarity matrix, using a quadratic re-ranking algorithm to perform image-to-text retrieval and text-to-image retrieval, and updating network parameters by dynamic triple loss back propagation.

2. The image-text matching method based on a graph neural network according to claim 1, characterized in that, The steps of obtaining the image representation are: extracting key regions of the image through target detection by using a Faster R-CNN network, and extracting the representation of the key regions by using a ResNet convolutional neural network, and combining the obtained regions to obtain the image representation.

3. The image-text matching method based on a graph neural network according to claim 1, characterized in that, The text representation is obtained by attention calculation, and the specific steps are: unifying the dimension of the word embedding vector of the text data and the dimension of the region embedding vector of the image data through dimension conversion; inputting the word embedding vector into a recurrent network to extract forward context word features and reverse context word features, and combining the forward context word features and the reverse context word features to obtain the text representation.

4. The image-text matching method based on a graph neural network according to claim 1, characterized in that, constructing a multi-modal knowledge graph training neural network model, and the steps comprise: Obtain the training images {I1,...,I N }, and extract triples {I} from the training images. i O i ,T i }, where I i It is the original image, O i It appears in I i The list of image objects in T i Image I i The text description; constructing a relationship set of image objects and text descriptions according to a triple, and calculating the co-occurrence times to obtain a co-occurrence matrix; calculating the embedding vector according to the co-occurrence matrix; calculating a text path similarity matrix according to the text description; calculating an object path similarity matrix according to the object list; the neural network combines the text path similarity matrix and the object path similarity matrix to reason the embedding vector, and updates the learnable matrix in the neural network according to the reasoning result until convergence.

5. The image-text matching method based on a graph neural network according to claim 1, characterized in that, The calculation steps of the local similarity comprise: obtaining image features in each region element and word features in each word element, and calculating the attention weight of each region element; calculating word visual features according to the attention weight of the region element, and calculating the similarity between the word visual features and the word features to obtain the participation similarity of each word feature; taking the participation similarity of each word feature as a node, calculating the edges between nodes, constructing a similarity graph, and reasoning each node and each edge to obtain the local similarity.

6. The image-text matching method based on a graph neural network according to claim 5, characterized in that, The reasoning of each node and each edge in the similarity graph comprises: the edge between any two nodes is calculated as: where S p and S q are two arbitrary nodes, W in and W out are linear transformations of incoming and outgoing nodes, respectively; the reasoning of the similarity graph is updated by updating the nodes and the edges in the following way: wherein, and are learnable parameters.

7. The image-text matching method based on a graph neural network according to claim 1, characterized in that, the calculation function of the dynamic triple loss is: I and respectively represent the query image and the query text, T, T' represent the positive sample text and the negative sample text of the query image, represent the positive sample image and the negative sample image of the query text.

8. The image-text matching method based on a graph neural network according to claim 1, characterized in that, The steps of searching by using the secondary rearrangement method include: inputting an image, performing first round searching according to similarity to obtain K first similar texts most similar to the input image, obtaining M similar images most similar to each first similar text, embedding a position index for the similar text according to the position of the similar image to obtain an image position set corresponding to the input image, and outputting a first similar text sequence after sorting the image position set; selecting a second similar text from the similar text sequence to perform second round searching, and outputting a second similar text sequence.

Images