A method for image text matching

Abstract

The present invention provides a method for image-text matching, comprising: using a deep neural network to extract multi-scale image features and text features respectively; using a self-attention mechanism method to calculate the global similarity of multiple scales; performing node-level matching and structure-level matching, and using the structure-level matching results to calculate the local similarity of multiple scales; calculating the final total similarity according to the calculated global similarity and local similarity of multiple scales; performing supervised learning model training, updating the model parameters according to the loss function, and obtaining the trained model; using the trained model to calculate the final total similarity of the input image and text, and obtaining the matched text or image according to the final total similarity size. The present invention can find matching images or texts more quickly on the premise that network training is completed; and obtain more accurate results by using the multi-scale features of the original data and a more comprehensive similarity measurement method.

Description

Technical Field

[0001] The present invention relates to the field of intelligent matching technology, and in particular to a method for matching one modality with another modality in image and text multimodal data. Background Art

[0002] In daily life, especially in the process of accessing Internet resources, images and texts are two very important modalities. Image search is usually inseparable from text description, and matching images to texts with similar semantic information also plays a guiding role in finding similar images. Therefore, image-text matching, that is, the method of calculating the similarity between images and texts, is very important. Traditional methods not only need to save a large amount of relational data in addition to the data itself, but also cannot cope well with dynamically changing data. Moreover, each calculation takes a long time and is not applicable to scenarios with real-time requirements. The image-text matching method based on deep learning extracts the features of images and texts respectively, aligns the features, and calculates the similarity. After supervised training with a large number of data sets, the obtained model can quickly and accurately find texts matching images or images matching texts in the data to be processed. At present, there are many technologies for deep learning methods of image-text matching, but basically they cannot fully consider the matching of the whole and the local, and few people pay attention to the situation where the size difference of key targets in the image is large.

[0003] Therefore, how to provide a method that can perform multi-scale feature matching and a more comprehensive similarity measurement matching between images and texts is an urgent problem that those skilled in the art need to solve. Summary of the invention

[0004] In view of this, the present invention proposes a method for extracting image features and text features based on deep learning and multimodal learning, and performing overall and local similarity matching.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] A method for image text matching includes a model building step and a model matching step:

[0007] The model building step includes:

[0008] S1. Using a deep neural network to extract multi-scale image features and text features respectively, the image and text are data with matching annotations;

[0009] S2. Use the self-attention mechanism to calculate the global similarity of multiple scales of image multi-scale features and text multi-scale features;

[0010] S3, performing node-level matching and structure-level matching between the image multi-scale features and the text multi-scale features in sequence, and using the structure-level matching results to calculate the local similarities of multiple scales;

[0011] S4, calculating the final total similarity according to the calculated global similarity and local similarity of multiple scales;

[0012] S5, repeat S2-S4 to perform supervised learning model training, update model parameters according to the loss function, and obtain the trained model;

[0013] Model matching steps:

[0014] S6. Calculate the final total similarity between the input image and text using the trained model, and obtain the matching text or image based on the final total similarity.

[0015] Preferably, S1 includes the step of extracting multi-scale features of an image:

[0016] Input the image into the Faster R-CNN model to identify the detection boxes of several image regions;

[0017] Encoding the image area corresponding to the detection frame using a ResNet model to obtain extracted visual features;

[0018] The visual features are respectively passed through a fully connected layer with an output dimension of N scales, and the output matrix is ​​the multi-scale feature of the image, N∈N+.

[0019] Preferably, S1 includes a text multi-scale feature extraction step:

[0020] Tokenize the sentence at word level by assigning each word a unique number to obtain the sentence vector;

[0021] After embedding the sentence vector into a multi-dimensional vector space, a text feature vector is obtained through a bidirectional GRU; N bidirectional GRUs are provided to output text feature vectors of N scales.

[0022] Preferably, the node-level matching in S3 is a step of matching an image with a word in a sentence, comprising:

[0023] The image feature T is fused with the text feature V, and then the softmax function is calculated along the visual axis to obtain the similarity matrix, that is, W = softmax(λTV T ), λ is the temperature parameter;

[0024] Using the similarity matrix as a weight parameter to integrate all image features into weighted image features;

[0025] The cosine similarity measurement method is used to calculate the similarity between weighted image features and text features to obtain node-level matching results.

[0026] Preferably, the structural level matching in S3 is a step of matching the image with the phrases in the sentence, including:

[0027] Analyze the semantic dependencies between words in the text sentence, construct the text features into an undirected sparse graph, and record the word nodes with dependencies as adjacent nodes;

[0028] Calculate the similarity matrix between word representations and obtain the weights of edges in the undirected sparse graph;

[0029] The graph convolutional neural network is used to integrate the neighborhood matching vectors represented by the similarity matrix and update the nodes in the similarity matrix.

[0030] Preferably, the calculation of local similarity using the structure-level matching result in S3 specifically includes: sending the result of the graph convolutional neural network to a fully connected layer and calculating the l2-norm to obtain the local similarity.

[0031] Preferably, S4 includes:

[0032] The global similarity and the local similarity are concatenated into a matrix and then sent to a fully connected layer to obtain a total similarity at a scale;

[0033] The total similarities at multiple scales are weighted summed to obtain the final total similarity.

[0034] Preferably, the model output of S6 is: a matrix with the number of rows equal to the number of images / texts and the corresponding number of columns equal to the number of texts / images, and each value in the matrix represents the similarity calculated between the image / text represented by the row where the value is located and the text / image represented by the column where the value is located.

[0035] It can be seen from the above technical solution that, compared with the prior art, the beneficial effects of the present invention include:

[0036] The present invention can find matching images or texts more quickly under the premise that network training is completed; compared with existing deep learning technologies, more accurate results can be obtained by utilizing multi-scale features of original data and a more comprehensive similarity measurement method. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on the provided drawings without creative work.

[0038] Figure 1 A flowchart of the image text matching method provided by an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of training data provided by an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of target detection in an image region provided by an embodiment of the present invention;

[0041] Figure 4 A schematic diagram of text matching results based on an image provided by an embodiment of the present invention;

[0042] Figure 5 A schematic diagram of image matching results based on text provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] See attached Figure 1 The present invention provides an image text matching method based on multi-scale feature extraction and multi-scale similarity calculation, which specifically includes a model building step and a model matching step:

[0045] The steps to build the model include:

[0046] S1. Use deep neural networks to extract multi-scale image features and text features respectively. Images and texts are matched and annotated data.

[0047] In the specific implementation of this step, the purpose of matching and annotating images and texts is to establish a training data set. The training data set reflects the training texts that are similar to the training images and is used for supervised training of the model. Each image contains 5 description sentences. For examples of the correspondence between images and texts, see the attached Figure 2 shown.

[0048] In one embodiment, the steps of extracting multi-scale features of an image are as follows:

[0049] Input the image into the Faster R-CNN model to identify the detection boxes of several image regions;

[0050] Use the ResNet model to encode the image area corresponding to the detection box to obtain the extracted visual features;

[0051] The visual features are passed through a fully connected layer with an output dimension of N scales, and the output matrix is ​​the multi-scale feature of the image, N∈N+.

[0052] In the specific implementation: For image features, the Faster R-CNN model pre-trained on Visual Genomes is used to identify 36 image regions containing entities and attributes, see Appendix Figure 3 As shown in the figure, the two sub-images above are the original image and all the recognized image regions, respectively. The six sub-images below are six samples selected from them. It can be seen that the recognized regions not only include entities such as shadows and hair, but also attributes such as color. Then the ResNet model is used to encode the image regions corresponding to the 36 detection frames to obtain the extracted visual features. In order to obtain multi-scale features, the visual features are passed through fully connected layers with output dimensions of 128, 256, 512, and 1024 (i.e., four scales), and the output matrix is ​​the multi-scale features of the image.

[0053] In one embodiment, the steps of extracting multi-scale features of text are as follows:

[0054] Tokenize the sentence at word level by assigning each word a unique number to obtain the sentence vector;

[0055] After embedding the sentence vector into a multi-dimensional vector space, a text feature vector is obtained through a bidirectional GRU; N bidirectional GRUs are set to output text feature vectors of N scales.

[0056] In specific implementation: For text features, the sentences need to be tokenized at the word level first. All the words in the data set are counted, each word is given a unique number, and a word-number index table is established. The sentences can be encoded into a digital matrix by replacing the words in the sentence with numbers. At the same time, since the RNN network is used later, it is also necessary to add the start and end corresponding numbers to the beginning and end of each sentence. After embedding each sentence vector into a 300-dimensional vector space, two encoded vectors are obtained through a bidirectional GRU. The average of these two vectors is obtained to obtain the text feature vector. In order to match the multi-scale features of the image, there are 4 bidirectional GRU networks, and the output dimensions are also 128, 256, 512 and 1024.

[0057] S2. Use the self-attention mechanism method to calculate the global similarity of multiple scales of multi-scale image features and multi-scale text features.

[0058] In one embodiment, for visual features and text features, the self-attention mechanism is first used to calculate the global feature vectors. The most common method for calculating the similarity between two vectors is to use the cosine similarity metric, but in order to enable the model to learn more complex matches, a similarity calculation method with parameters is used, namely x and y represent visual features and text features respectively, W is a learnable parameter, |·| 2 represents element-wise square, ‖·‖2 represents the l2-norm, and the result is the global similarity.

[0059] S3. Perform node-level matching and structure-level matching between the multi-scale features of the image and the multi-scale features of the text in turn, and use the structure-level matching results to calculate the local similarities of multiple scales.

[0060] In this step, the local similarity calculation can be divided into two parts: node-level matching and structure-level matching.

[0061] In one embodiment, node-level matching is a step of matching an image with a word in a sentence, including:

[0062] The image feature T is fused with the text feature V, and then the softmax function is calculated along the visual axis to obtain the similarity matrix, that is, W = softmax(λTV T ), λ is the temperature parameter, and the obtained similarity matrix result can measure the similarity of the image to each text;

[0063] The similarity matrix is ​​used as a weight parameter to integrate all image features into weighted image features;

[0064] The cosine similarity measurement method is used to calculate the similarity between weighted image features and text features to obtain node-level matching results.

[0065] In one embodiment, structure-level matching is a step of matching an image with a phrase in a sentence. Structure-level matching takes the result of node-level matching as input to learn fine-grained phrase correspondences. It includes:

[0066] The Stanford CoreNLP toolkit is used to parse the semantic dependencies between words in a sentence, and the text features are constructed as an undirected sparse graph. Word nodes with dependencies are recorded as adjacent nodes. A matrix is ​​used to represent the adjacent matrix of each node, and each node is adjacent to itself.

[0067] By using the softmax function, we can calculate the similarity matrix between word representations and record it as sij =softmax(λu i u j ),u i 、u j Represent words i and j respectively, and get the weight of the edge in the undirected sparse graph. The weight of the edge in the sparse graph can be obtained by the l2-norm;

[0068] The graph convolutional neural network is used to integrate the neighborhood matching vectors represented by the similarity matrix and update the nodes in the similarity matrix.

[0069] In one embodiment, calculating local similarity using the structure-level matching result specifically includes: sending the result of the graph convolutional neural network to a fully connected layer and calculating the l2-norm to obtain the local similarity.

[0070] S4. Calculate the final total similarity based on the calculated global similarities and local similarities at multiple scales.

[0071] In one embodiment, the global similarity and the local similarity are concatenated into a matrix and then sent to the fully connected layer to obtain the total similarity at one scale. The concatenation rule is to add the (1,n)-dimensional vector to the (n,n)-dimensional matrix to form an (n+1,n)-dimensional matrix; the total similarities at multiple scales are weighted and summed to obtain the final total similarity. In this embodiment, the weights assigned to the four scales are 0.125, 0.25, 0.5 and 1 respectively.

[0072] S5. Repeat S2-S4 to perform supervised learning model training, update the model parameters according to the loss function, and obtain the trained model.

[0073] The loss function used in this embodiment is triple loss. When the query type is image I, T is sampled from the matched text and T' is sampled from the unmatched text; symmetrically, when the query is text T, the matched image I and the unmatched image I' are sampled. In order to minimize the loss function, the similarity score of the matched image-text pair can be high and the similarity score of the unmatched image-text pair can be low. The loss function is designed as follows:

[0074]

[0075] S(.,.) represents the similarity calculation model established in the embodiment of the present invention, [·] +Indicates that the expression is subjected to the max(·,0) operation, α is the marginal value, set to 0.2. The method adopted in this embodiment is supervised learning, so a large amount of supervised data is required for training, that is, data with matching annotations for images and texts. Each time steps 2 to 4 are repeated, a portion of the data is processed according to the size of the GPU video memory of the training machine, and the model parameters are updated in each cycle according to the back propagation of the loss function. After all the data is sent into the network once, it is recorded as a cycle, and the training is stopped until the specified number of cycles is reached. In this embodiment, the number of images processed in each training is set to 64, and the training lasts for 40 rounds in total. A dynamic learning rate is used to prevent unstable learning. The loss function is 0.002 in the first 30 rounds, and the loss function is set to 0.0002 in the last 10 rounds. After each round of training, the model will be verified on a small annotated data set (validation set), and the best performing model on the validation set is selected as the final training result after the training is completed.

[0076] Model matching steps:

[0077] S6. Calculate the final total similarity between the input image and text using the trained model, and obtain the matching text or image based on the final total similarity.

[0078] In one embodiment, testing the model is to simulate the performance of the model in actual use. Images and texts outside the training set are selected as model inputs, and the model output is: a matrix with the number of rows equal to the number of images / texts and the corresponding number of columns equal to the number of texts / images. Each value in the matrix represents the calculated similarity between the image / text represented by the row where the value is located and the text / image represented by the column where the value is located. If you want to calculate the text that matches a certain image, you can sort the rows corresponding to the images in the matrix, and the text represented by the column with the largest value is the calculated best matching text. Correspondingly, you can also calculate the image that matches a certain text.

[0079] In order to prove the superiority of the matching effect of the present invention, the following is a comparative explanation based on specific image-text matching results:

[0080] Attached Figure 4 These are two examples of matching texts based on images. The five sentences respectively represent the texts that are calculated to be most similar to the image. It can be seen that, using the matching results of the present invention, the highest similarity is successfully calculated with the correct text for both images. The fifth sentence of the wrong second image also proves that the present invention successfully captures the entity "dog".

[0081] Attached Figure 5These are two examples of matching images based on text. Because there is only one image corresponding to one sentence in the annotation of the dataset, the image corresponding to the annotation is marked as correct, and the other four images are marked as wrong. However, it can be found that "toilet", "bathroom", "dirtbike" or "wine bottle" correspond to the text in the other images.

[0082] The image-text matching method provided by the present invention is introduced in detail above. In this embodiment, specific examples are used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0083] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined in the present embodiments may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown in the present embodiment, but will conform to the widest range consistent with the principles and novel features disclosed in the present embodiment.

Claims

1. A method for image text matching, characterized in that: Including model building steps and model matching steps: The model building step includes: S1. Using a deep neural network to extract multi-scale image features and text features respectively, the image and text are data with matching annotations; S2. Use the self-attention mechanism to calculate the global similarity of multiple scales of image multi-scale features and text multi-scale features; S3, performing node-level matching and structure-level matching between the multi-scale features of the image and the multi-scale features of the text in sequence, and using the structure-level matching results to calculate the local similarities of multiple scales; the structure-level matching is a step of matching the image with the phrases in the sentence, including: Structure-level matching takes the results of node-level matching as input to learn fine-grained phrase correspondences; Analyze the semantic dependencies between words in the text sentence, construct the text features into an undirected sparse graph, and record the word nodes with dependencies as adjacent nodes; Calculate the similarity matrix between word representations and obtain the weights of edges in the undirected sparse graph; Use graph convolutional neural network to integrate the neighborhood matching vectors represented by the similarity matrix and update the nodes in the similarity matrix; S4, calculating the final total similarity according to the calculated global similarity and local similarity of multiple scales; S5, repeat S2-S4 to perform supervised learning model training, update model parameters according to the loss function, and obtain the trained model; Model matching steps: S6. Calculate the final total similarity between the input image and text using the trained model, and obtain the matching text or image based on the final total similarity.

2. The image-text matching method according to claim 1, characterized in that: The S1 includes the steps of extracting multi-scale features of an image: Input the image into the Faster R-CNN model to identify the detection boxes of several image regions; Encoding the image area corresponding to the detection frame using a ResNet model to obtain extracted visual features; The visual features are respectively passed through a fully connected layer with an output dimension of N scales, and the output matrix is ​​the multi-scale feature of the image, N∈N+.

3. The image-text matching method according to claim 1, characterized in that: S1 includes the steps of text multi-scale feature extraction: Tokenize the sentence at word level by assigning each word a unique number to obtain the sentence vector; After embedding the sentence vector into a multi-dimensional vector space, a text feature vector is obtained through a bidirectional GRU; N bidirectional GRUs are provided to output text feature vectors of N scales.

4. The image-text matching method according to claim 1, characterized in that: The node-level matching in S3 is a step of matching an image with a word in a sentence, including: The image feature T is fused with the text feature V, and then the softmax function is calculated along the visual axis to obtain the similarity matrix, that is, W = softmax(λTV T ), λ is the temperature parameter; Using the similarity matrix as a weight parameter to integrate all image features into weighted image features; The cosine similarity measurement method is used to calculate the similarity between weighted image features and text features to obtain node-level matching results.

5. The image-text matching method according to claim 1, characterized in that: The calculation of local similarity using the structure-level matching result in S3 specifically includes: sending the result of the graph convolutional neural network to the fully connected layer and calculating the l2-norm to obtain the local similarity.

6. The image-text matching method according to claim 1, characterized in that: The S4 includes: The global similarity and the local similarity are concatenated into a matrix and then sent to a fully connected layer to obtain a total similarity at a scale; The total similarities at multiple scales are weighted summed to obtain the final total similarity.

7. The image-text matching method according to claim 1, characterized in that: The model output of S6 is: a matrix with the number of rows equal to the number of images / texts and the corresponding number of columns equal to the number of texts / images. Each value in the matrix represents the similarity calculated between the image / text represented by the row where the value is located and the text / image represented by the column where the value is located.

Images