Image-text cross-modal retrieval method based on memory unit optimization features

Abstract

The present invention discloses a method for mutual retrieval of images and texts based on optimized features of memory units, which mainly solves the problem that the existing technology is difficult to accurately express the semantic features of images and texts, resulting in low retrieval accuracy. Its implementation scheme is as follows: obtain a training set and a test set; construct a feature extraction network and a memory unit sub-network respectively; train the feature extraction network with the training set, and train the memory unit sub-network with the image and text features output by the network; input the test set into the trained feature extraction network to obtain the features of the image and text; input the features of the image and text into the optimized memory unit sub-network to obtain the optimized image features and the optimized text features; calculate the feature similarity between the optimized image and the optimized text, and retrieve the image-text retrieval results and the text-image retrieval results. The present invention can accurately express the semantic features of images and texts, improve the retrieval accuracy, and can be used for the mutual matching of natural images and texts.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and further relates to a cross-modal image-text retrieval method that can be used for mutual matching between natural images and text. Background Art

[0002] Image-text cross-modal retrieval is to retrieve semantically identical image-text pairs from natural images and texts. There are two main existing image-text cross-modal retrieval methods: one is the global feature retrieval method based on images and texts, and the other is the local feature retrieval method based on images and texts. The global feature retrieval method based on images and texts is to construct a feature extraction network to extract feature vectors of the entire image and the entire text, take these features as their global features, use a metric method to calculate the similarity between features, and select the image-text pair with the highest similarity as the retrieval result. The local feature retrieval method based on images and texts is to construct a feature extraction network to extract feature vectors of image regions and text words, take these features as their local features, use a metric method to obtain the similarity of image region-word pairs, and further aggregate to obtain the similarity of image-text pairs, and select the image-text pair with the highest similarity as the retrieval result.

[0003] T. Wang, X. Xu, Y. Yang, et al. proposed an image-text retrieval method based on multimodal fusion vectors in their paper “Matching images and text with multi-modal tensor fusion and re-ranking” (ACM International Conference on Multimedia, 2019: 12-20). It extracts image features and text features by constructing a multimodal feature fusion network, calculates feature similarity through a measurement method, and re-ranks the retrieval results by leveraging the bidirectionality of image-text retrieval. Although this method uses a multi-layer network to extract high-order features of images and texts, and obtains image-text similarity through high-order features to achieve the retrieval task, since the method processes the entire image and the entire text, it is easily affected by redundant information such as background information during the feature extraction process, making it difficult to measure the similarity between image and text features, thus affecting the retrieval accuracy.

[0004] In their paper "Stacked cross attention for image-text matching" (European Conference on Computer Vision, 2018: 201-216), K. Lee, X. Chen, G. Hua, et al. proposed a stacked attention-based image-text retrieval method. This method uses an object detection algorithm to obtain image regions, constructs a feature extraction network to obtain features of the image regions and features of text words, uses an attention mechanism to highlight important information in the features, and uses this information to calculate the similarity between the image and text, thus achieving image-text retrieval. Although this method eliminates the impact of redundant global feature information to a certain extent by training local features, it only uses the attention mechanism to explore the semantic information between the image and text when extracting image and text features, ignoring the internal relationship between the image region and the text words. Therefore, it cannot accurately express the semantic features of the image and text, affecting retrieval accuracy. Summary of the Invention

[0005] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose a cross-modal image-text retrieval method based on memory unit optimization features to accurately express the semantic features of images and texts and improve the retrieval accuracy.

[0006] The idea of ​​achieving the purpose of the present invention is to fully explore the semantic information within the image and text modalities by extracting the features of images and texts using the self-attention structure; complete the feature interaction between images and texts through the learnable parameter layer in the memory unit subnetwork; and store the learned interaction information between images and texts during the training process so that the final features contain richer semantic information.

[0007] According to the above ideas, the implementation steps of the present invention include the following:

[0008] (1) Randomly select 80% of the images and their corresponding text pairs from the Flickr30k dataset to form the training set, and 20% of the images and their corresponding text pairs to form the test set;

[0009] (2) Constructing feature extraction network:

[0010] (2a) Build an image feature processing subnetwork consisting of a first convolutional layer, a second convolutional layer, and a fully connected layer in cascade to extract image features;

[0011] (2b) Build a text feature processing subnetwork consisting of a cascade of Bert layers and fully connected layers to extract text features;

[0012] (2c) connecting the image feature processing subnetwork and the text feature processing subnetwork in parallel to form a feature extraction network;

[0013] (3) Randomly select 1% of the unselected images and their corresponding texts from the training set as the input of the feature extraction network;

[0014] (4) Input the selected images and their corresponding texts into the feature extraction network, and output the features of each image and the features of the text corresponding to each image respectively;

[0015] (5) Construct a memory unit subnetwork consisting of a cascade of a learnable parameter layer, a first self-attention layer, a second self-attention layer, a third self-attention layer, and a fully connected layer;

[0016] (6) Input the features of each image and its corresponding text into the memory unit subnetwork, and output the optimized features of each image and the optimized features of the text corresponding to each image respectively;

[0017] (7) Calculate the feature similarity S(i, j) between each image and its corresponding text in the selected image and its corresponding text, where i represents the i-th image and j represents the j-th text;

[0018] (8) According to the feature similarity S(i,j), set the loss function of the memory unit sub-network:

[0019] Loss′(i,j)=max[0.2-S(i,j)+S(i,·) [n] ,0]+max[0.2-S(i,j)+S(·,j) [n] ,0]

[0020] Among them, Loss′(i,j) represents the loss value of the features of the i-th image and the j-th text in the selected image and its corresponding text; S(i,·) [n] Indicates the calculation of the feature similarity between the i-th image and all texts in the selected image and its corresponding text, and the feature similarity of the n-th position selected after sorting from large to small; S(·,j) [n] It means calculating the feature similarity between the jth text in the selected image and its corresponding text and all images, and the feature similarity of the nth position selected after sorting from large to small;

[0021] (9) Substitute the feature similarity between each image and its corresponding text into the loss function Loss′(i, j) to obtain the loss value of each image and its corresponding text, and use it to calculate the gradient of each node of the feature extraction network and the memory unit subnetwork;

[0022] (10) Use the gradient to update the weights of each node in the feature extraction network and the memory unit sub-network, and determine whether the loss value of the memory unit sub-network after the current update iteration is less than 0.00001:

[0023] If so, the trained feature extraction network and memory unit sub-network are obtained, and step (11) is executed;

[0024] Otherwise, return to step (3);

[0025] (11) Mutual retrieval of the image and text to be retrieved:

[0026] (11a) Input all image-text pairs in the test set into the trained feature extraction network to obtain the features of each image and each text in the image-text pair to be retrieved;

[0027] (11b) Input the features of each image and each text into the trained memory unit sub-network to obtain the optimized features of each image and each text;

[0028] (11c) Calculate the feature similarity S(i,j) between each optimized image and each optimized text;

[0029] (11d) From all the calculated similarities, select the text with the highest similarity between each image and itself as the image-text retrieval result, and select the image with the highest similarity between each text and itself as the text-image retrieval result.

[0030] Compared with the prior art, the present invention has the following advantages:

[0031] First, by using a memory unit sub-network to optimize the features of images and texts, the present invention overcomes the problem in the prior art that only using the attention mechanism cannot fully exploit the interactive information between different modalities of images and texts, thereby improving the accuracy of image and text matching.

[0032] Second, since the present invention sets the loss function of the memory unit sub-network to perform gradient updates on the feature extraction network and the memory unit sub-network, the restriction conditions of the traditional triple ranking function are relaxed, and the problem of overfitting caused by the overly strict conditions of the triple ranking loss in the prior art is overcome. The adverse effects of the overfitting problem on the expression of semantic features of images and texts are reduced, and the semantic features of images and texts can be accurately expressed, thereby further improving the retrieval accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Flowchart for the implementation of the present invention;

[0034] Figure 2 Schematic diagram of the network structure in the present invention;

[0035] Figure 3 Schematic diagram of the memory unit sub-network structure in the present invention. DETAILED DESCRIPTION

[0036] The embodiments of the present invention are further described below with reference to the accompanying drawings.

[0037] Reference Figure 1 , the implementation steps for the instance are as follows:

[0038] Step 1: Get the training set and test set.

[0039] 80% of the images and their corresponding text pairs are randomly selected from the publicly available Flickr30k dataset to form the training set, and 20% of the images and their corresponding text pairs are selected to form the test set.

[0040] This example selects but is not limited to 26,783 images and their corresponding text pairs from the Flickr30k dataset to form a training set, and 5,000 images and their corresponding text pairs to form a test set.

[0041] Step 2: Build a feature extraction network.

[0042] Reference Figure 2 , the present invention is implemented as follows:

[0043] 2.1) Build an image feature processing subnetwork consisting of a first convolutional layer, a second convolutional layer, and a fully connected layer in cascade to extract image features:

[0044] The convolution kernel size of the first convolution layer and the second convolution layer are both set to 3*3, and the step size is both set to 1;

[0045] The number of input nodes and output nodes of the fully connected layer is 2048 and 1024 respectively;

[0046] 2.2) Build a text feature processing subnetwork consisting of a cascade of Bert layers and fully connected layers to extract text features:

[0047] The Bert layer is composed of 12 self-attention layers in series, the number of multi-heads of all self-attention layers is set to 12, and the number of output nodes of all self-attention layers is set to 768;

[0048] The number of input nodes and output nodes of the fully connected layer is 768 and 1024 respectively;

[0049] 2.3) The image feature processing subnetwork and the text feature processing subnetwork are connected in parallel to form a feature extraction network.

[0050] Step 3: Randomly select 1% of the unselected images and their corresponding texts from the training set as input to the feature extraction network. In this example, 267 unselected images and their corresponding texts are selected but not limited to.

[0051] In step 4, the selected images and their corresponding texts are input into the feature extraction network, and the features of each image and the features of the text corresponding to each image are output respectively.

[0052] Step 5: Build the memory unit sub-network.

[0053] Reference Figure 3 , the implementation of the steps is as follows:

[0054] 5.1) Set up a learnable parameter layer, which contains one variable parameter of size 2*1024. Use the following formula to concatenate the variable parameter for each image and its corresponding text:

[0055]

[0056]

[0057] Among them, i represents the i-th image, j represents the j-th text, M represents the variable parameter, v i represents the features of the i-th image in the selected image and its corresponding text, Represents the features of the i-th image in the selected image and its corresponding text after splicing variable parameters, t j Represents the features of the jth text in the selected image and its corresponding text, represents the features of the jth text in the selected image and its corresponding text after splicing with variable parameters, and [·,·] represents the vector splicing operation;

[0058] 5.2) Set the number of heads in the three self-attention layers to 12 and the number of output nodes to 1024;

[0059] 5.3) Set the number of input nodes and output nodes of the fully connected layer to 1024;

[0060] 5.4) The learnable parameter layer, the first self-attention layer, the second self-attention layer, the third self-attention layer, and the fully connected layer are cascaded in sequence to form a memory unit sub-network.

[0061] Step 6: Obtain the feature similarity S(i, j) between each image and its corresponding text.

[0062] 6.1) Input the features of each image and its corresponding text into the memory unit subnetwork, and output the optimized features of each image and the optimized features of each image and its corresponding text respectively;

[0063] 6.2) Calculate the feature similarity S(i,j) between each image and its corresponding text:

[0064]

[0065] in, represents the optimized features of the i-th image in the selected image and its corresponding text, represents the multiplication operation, represents the optimized features of the j-th text in the selected image and its corresponding text, and ||·|| represents the modulo operation.

[0066] Step 7: Set the loss function of the memory unit sub-network.

[0067] This step sets the loss function of the memory unit subnetwork according to the feature similarity S(i,j) and the existing triple ranking loss:

[0068] The existing triplet ranking loss is expressed as follows:

[0069] Loss(i,j)=max[0.2-S(i,j)+S(i,k),0]+max[0.2-S(i,j)+S(k,j),0]

[0070] Among them, Los(i,j) represents the loss value of the features of the i-th image and the j-th text in the selected image and its corresponding text, S(i,k) represents the calculation of the feature similarity between the i-th image and the k-th text in the selected image and its corresponding text, and S(k,j) represents the calculation of the feature similarity between the k-th image and the j-th text in the selected image and its corresponding text;

[0071] In the triple ranking loss, the feature similarity S(i,j) must be greater than the feature similarities of all unmatched image-text pairs. This restriction is too strict and may cause overfitting. Therefore, this example relaxes the restriction and sets the loss function of the memory unit subnetwork based on the triple ranking loss. The implementation is as follows:

[0072] 7.1) Calculate the feature similarity between the i-th image and all texts in the selected image and its corresponding text, sort them from large to small, and select the feature similarity S(i,·) at the n-th position. [n] , using this S(i,·) [n] Replace the feature similarity S(i,k) between the i-th image and the k-th text;

[0073] 7.2) Calculate the feature similarity between the jth text in the selected image and its corresponding text and all images, sort them from large to small, and select the feature similarity S(·,j) at the nth position. [n], using this S(·,j) [n] Replace the feature similarity S(k,j) between the k-th image and the j-th text;

[0074] 7.3) According to the results of 7.1) and 7.2), the loss function Loss′(i,j) of the memory unit sub-network is obtained:

[0075] Loss′(i,j)=max[0.2-S(i,j)+S(i,·) [n] ,0]+max[0.2-S(i,j)+S(·,j) [n] ,0].

[0076] Step 8: Calculate the gradient of each node in the feature extraction network and memory unit sub-network.

[0077] 8.1) Substitute the feature similarity between each image and its corresponding text into the loss function Loss′(i, j) of the memory unit subnetwork to obtain the loss value θ(i, j) for each image and its corresponding text;

[0078] 8.2) Based on the loss value θ(i, j) for each image and its corresponding text, the gradient of each node in the feature extraction network and the memory unit subnetwork is calculated using the following formula:

[0079]

[0080] Among them, Δ(w k ) represents the gradient value of the kth node weight in all nodes of the feature extraction network or memory unit subnetwork, w k Represents the k-th node weight among all nodes in the feature extraction network or memory unit subnetwork.

[0081] Step 9: Update the weights of each node in the feature extraction network and memory unit sub-network.

[0082] 9.1) Using the gradient Δ(w k ) Use the following formula to update the weights of each node in the feature extraction network and the memory unit sub-network respectively:

[0083]

[0084] in, represents the updated weight of the kth node in all nodes of the feature extraction network or memory unit subnetwork, and α represents the learning rate with a value of 0.001;

[0085] 9.2) Determine whether the loss value of the memory unit subnetwork after the current update iteration is less than 0.00001:

[0086] If so, the trained feature extraction network and memory unit sub-network are obtained, and step 10 is executed;

[0087] Otherwise, return to step 3.

[0088] Step 10: Use the trained feature extraction network and memory unit sub-network to perform mutual retrieval on the image text to be retrieved.

[0089] 10.1) Input all image-text pairs in the test set into the trained feature extraction network to obtain the features of each image and each text in the image-text pair to be retrieved;

[0090] 10.2) Input the features of each image and each text into the trained memory unit subnetwork to obtain the optimized features of each image and each text;

[0091] 10.3) Calculate the feature similarity S(i, j) between each optimized image and each optimized text;

[0092] 10.4) From all the calculated similarities, select the text with the highest similarity for each image as the image-text retrieval result, and select the image with the highest similarity for each text as the text-image retrieval result, completing the image-text cross-modal retrieval.

[0093] The effects of the present invention are further illustrated by the following simulation experiments.

[0094] 1. Simulation experiment conditions:

[0095] The hardware platform of the simulation experiment of the present invention is: Intel(R) Core(TM) i7-8700K CPU, 3.70GHz*12, NVIDIA GeForce GTX 1080Ti, and 64G memory.

[0096] The software platform for the simulation experiment of the present invention is: Pytorch.

[0097] The dataset used in this simulation experiment is the Flickr30k dataset collected by Yahoo, which has a total of 31,783 images. Each image in this dataset is described by 5 texts. The training set of this dataset has 26,783 images and the test set has 5,000 images.

[0098] 2. Simulation content and result analysis:

[0099] Under the above conditions, the present invention and the existing image-text mutual retrieval method based on the self-attention encoding network TERAN are used to retrieve the text corresponding to each image and the image corresponding to each text in a test set containing 5000 natural image and text pairs, and the retrieval accuracy of the two retrieval methods is evaluated. The proportion of correct image-text retrieval results and text-image retrieval results in the entire test data are counted respectively. The results are shown in Table 1.

[0100] Table 1 Comparison of mutual retrieval accuracy between the present invention and the prior art in simulation experiments

[0101] Simulation Experiment Method Image-text retrieval Text-Image Retrieval TERAN 79.2 63.1 Method of the present invention 83.7 64.4

[0102] The existing method TERAN in Table 1 is the image-text mutual retrieval method proposed by N. Messina et al. in "Fine-grained visual textual alignment for cross-modal retrieval using transformer encoders. In: ACM Transactions on Multimedia Computing, Communications, and Applications, pp. 1741-1764, 2020", referred to as the image-text mutual retrieval method based on the self-attention encoding network TERAN.

[0103] As can be seen from Table 1, the method proposed in the present invention significantly improves the retrieval accuracy. The most obvious improvement is in the image-text retrieval task. Compared with the existing TERAN method, its retrieval accuracy is improved by 4.5%. This shows that the memory unit subnetwork constructed in the present invention can accurately express the semantic features of images and texts, further improving the retrieval accuracy.

Claims

1. A cross-modal image-text retrieval method based on memory unit optimization features, characterized in that: The steps include: (1) Randomly select 80% of the images and their corresponding text pairs from the Flickr30k dataset to form the training set, and 20% of the images and their corresponding text pairs to form the test set; (2) Constructing feature extraction network: (2a) Build an image feature processing subnetwork consisting of a first convolutional layer, a second convolutional layer, and a fully connected layer in cascade to extract image features; (2b) Build a text feature processing subnetwork consisting of a cascade of Bert layers and fully connected layers to extract text features; (2c) connecting the image feature processing subnetwork and the text feature processing subnetwork in parallel to form a feature extraction network; (3) Randomly select 1% of the unselected images and their corresponding texts from the training set as the input of the feature extraction network; (4) Input the selected images and their corresponding texts into the feature extraction network, and output the features of each image and the features of the text corresponding to each image respectively; (5) Construct a memory unit subnetwork consisting of a learnable parameter layer, a first self-attention layer, a second self-attention layer, a third self-attention layer, and a fully connected layer; the learnable parameter layer contains one variable parameter, the size of the variable parameter is 2*1024, and the variable parameter is spliced ​​for each image and its corresponding text using the following formula: Among them, M represents a variable parameter, v i Represents the features of the i-th image in the selected image and its corresponding text, Represents the features of the i-th image in the selected image and its corresponding text after splicing variable parameters, t j Represents the features of the jth text in the selected image and its corresponding text, represents the features of the jth text in the selected image and its corresponding text after splicing with variable parameters, and [·,·] represents the vector splicing operation; (6) Input the features of each image and its corresponding text into the memory unit subnetwork, and output the optimized features of each image and the optimized features of the text corresponding to each image respectively; (7) Calculate the feature similarity S(i, j) between each image and its corresponding text in the selected image and its corresponding text, where i represents the i-th image and j represents the j-th text; (8) According to the feature similarity S(i,j), set the loss function of the memory unit sub-network: Loss′(i,j)=max[0.2-S(i,j)+S(i,·) [n] ,0]+max[0.2-S(i,j)+S(·,j) [n] ,0] Among them, Loss′(i,j) represents the loss value of the features of the i-th image and the j-th text in the selected image and its corresponding text; S(i,·) [n] Indicates the calculation of the feature similarity between the i-th image and all texts in the selected image and its corresponding text, and the feature similarity of the n-th position selected after sorting from large to small; S(·,j) [n] It means calculating the feature similarity between the jth text in the selected image and its corresponding text and all images, and the feature similarity of the nth position selected after sorting from large to small; (9) Substitute the feature similarity between each image and its corresponding text into the memory unit sub-network loss function Loss′(i, j) to obtain the loss value θ(i, j) between each image and its corresponding text, and use it to calculate the gradient of each node of the feature extraction network and the memory unit sub-network; (10) Use the gradient to update the weights of each node in the feature extraction network and the memory unit sub-network, and determine whether the loss value of the memory unit sub-network after the current update iteration is less than 0.00001: If so, the trained feature extraction network and memory unit sub-network are obtained, and step (11) is executed; Otherwise, return to step (3); (11) Mutual retrieval of the image and text to be retrieved: (11a) Input all image-text pairs in the test set into the trained feature extraction network to obtain the features of each image and each text in the image-text pair to be retrieved; (11b) Input the features of each image and each text into the trained memory unit sub-network to obtain the optimized features of each image and each text; (11c) Calculate the feature similarity S(i, j) between each optimized image and each optimized text; (11d) From all the calculated similarities, select the text with the highest similarity between each image and itself as the image-text retrieval result, and select the image with the highest similarity between each text and itself as the text-image retrieval result.

2. The method according to claim 1, characterized in that The image feature processing subnetwork constructed in step (2a) has the following parameters for each layer: The convolution kernel size of the first convolution layer and the second convolution layer are both set to 3*3, and the step size is both set to 1; The number of input nodes and output nodes of the fully connected layer is 2048 and 1024 respectively.

3. The method according to claim 1, characterized in that The text feature processing subnetwork constructed in step (2b) has the following structure and parameters: The Bert layer is composed of 12 self-attention layers in series, the number of multi-heads of all self-attention layers is set to 4 to 12, and the number of output nodes of all self-attention layers is set to 768; The number of input nodes and output nodes of the fully connected layer is 768 and 1024 respectively.

4. The method according to claim 1, wherein The structure and parameters of the memory unit subnetwork constructed in step (5) are as follows: The number of multiple heads in the first, second and third self-attention layers is set to 4 to 12, and the number of output nodes is set to 1024. The number of input nodes and output nodes of the fully connected layer is set to 1024.

5. The method according to claim 1, wherein In step (7), the feature similarity S(i, j) between the i-th image and the j-th text is calculated using the following formula: in, represents the optimized features of the i-th image in the selected image and its corresponding text, represents the multiplication operation, represents the optimized features of the j-th text in the selected image and its corresponding text, and ||·|| represents the modulo operation.

6. The method according to claim 1, wherein In step (9), the gradients of the feature extraction network and the memory unit sub-network are calculated using the following formula: Among them, Δ(w k ) represents the gradient value of the kth node weight in all nodes of the feature extraction network or memory unit subnetwork, w k Represents the k-th node weight among all nodes in the feature extraction network or memory unit subnetwork.

7. The method according to claim 1, characterized in that In step (10), the weights of the feature extraction network and the memory unit sub-network are updated using the following formula: in, represents the updated weight of the kth node in all nodes of the feature extraction network or memory unit subnetwork, w k represents the kth node weight among all nodes in the feature extraction network or memory unit subnetwork, Δ(w k ) represents the gradient value of the kth node weight among all nodes in the feature extraction network or memory unit subnetwork, and α represents the learning rate with a value of 0.01 to 0.00001.

Images