Phrase matching combination-based cross-modal retrieval method

Abstract

The invention discloses a phrase matching combination-based cross-modal retrieval method, which comprises the following steps of: firstly, outputting a group of grid characteristics for each image to represent the image, and mapping the image to a public semantic space; then, converting the word sequence into text representation by using a text backbone network, and mapping the text representation to a public semantic space; thirdly, introducing different levels of GCN stacks, modeling non-neighborhood phrases in different ranges, and predicting image-text semantic similarity in a complex scene; then, through combination of local fragment matching, neighborhood phrase matching and multiple non-neighborhood phrase matching, the semantic similarity of the image-text is reflected more comprehensively; and finally, training by adopting a hinge loss supervision model with difficult negative sample mining. According to the method disclosed by the invention, cross-modal retrieval can be well carried out, and a competitive result is obtained from a plurality of databases.

Description

Technical Field

[0001] The present invention relates to a cross-modal retrieval method, and more particularly to a cross-modal retrieval method based on phrase matching combination. Background Art

[0002] With the rapid development of multimedia, there is a large amount of information on the Internet, such as images, texts, videos, audios, etc. It has become increasingly difficult to manually obtain useful information between different modalities in the vast amount of data. Naturally, we need a powerful method to help us obtain the required text, image or video. Cross-modal retrieval uses one modality of data as a query to retrieve relevant data of another modality. For example, we can use text to retrieve images of interest (just like what we do on Google Image Search), or use images to retrieve corresponding texts. Of course, the modalities are not limited to images and texts, and other modalities such as speech, physical signals and videos can also be an integral part of cross-modal retrieval. Existing cross-modal retrieval methods face the following problems in complex scenarios: First, the method based on global features ignores the local alignment between image regions and text words, and it is difficult to capture fine-grained semantic associations; Second, although the local fragment matching method pays attention to the correspondence between regions and words, it fails to model the multi-level relationships between fragments (such as neighborhood phrases and non-neighborhood phrases), resulting in insufficient sensitivity to the combined semantic differences (such as the confusion between "a person leading a horse" and "a person riding a horse"); Third, existing graph convolutional networks mostly adopt a single-layer structure, only covering local neighborhood relationships, lacking the ability to model non-neighborhood phrases in different ranges by stacking multiple layers of networks, which limits the hierarchical representation of complex scene semantics. These problems together lead to significant limitations in the existing methods in terms of local fragment relationship reasoning and the comprehensiveness of cross-modal semantic alignment. Summary of the Invention

[0003] To better perform cross-modal retrieval, the present invention provides a cross-modal retrieval method based on phrase matching combination.

[0004] The object of the present invention is achieved by the following technical solutions:

[0005] A cross-modal retrieval method based on phrase matching combination includes the following steps:

[0006] Step 1: For the image part in the image-text pair, use the pre-trained ResNeXT network to extract the grid features V′ = {v1′,..., v j ′,..., v n ′} of the image I, where: v j ′ represents the feature vector of the j-th grid after encoding, j = 1,..., n, and n represents the number of image grids; apply a linear mapping in the image grid feature space to map the grid features to the common semantic space to obtain the image feature set V = {v1,..., vj , …, v n},v j is the image grid feature vector in the common semantic space;

[0007] Step 2: For the text part in the image-text pair, use the text backbone network to convert the word sequence into a text representation U′ = {u1′, …, u i ′, …, u m ′}, where: u i ′ represents the feature vector of the i-th text word after encoding, i = 1, …, m, and m represents the number of words in the sentence; Apply a linear mapping in the text feature space to map the grid features to the common semantic space, and obtain the text feature set U = {u1, …, u i , …, u m}, u i is the text word feature vector in the common semantic space;

[0008] Step 3: Calculate the fine-grained similarity matrix S using the image feature set and the text feature set; With the help of the fine-grained similarity matrix, each image grid feature vector v j is represented by the weighted sum of all text features, and the corresponding image grid context feature vector is obtained Similarly, each text word feature vector u i is represented by the weighted sum of all image grid features, and the corresponding text word context feature vector is obtained

[0009] Step 4: Calculate the visual local segment matching feature V 1 , and perform local segment matching Calculate the text local segment matching feature U 1 using the text word feature set and the text word context feature set, and perform local segment matching

[0010] Step 5: Using the local segment matching features, calculate the neighborhood phrase matching feature and the non-neighborhood phrase matching feature respectively by stacking GCN, and perform neighborhood phrase matching and non-neighborhood phrase matching respectively;

[0011] Step 6: Combine the phrase matching results in Step 5, calculate the similarity s(I, T) between the image and the text, and perform supervision using the hinge loss with hard negative sample mining.

[0012] Compared with the prior art, the present invention has the following advantages:

[0013] 1. The present invention introduces the stacking of different levels of GCNs to model non-neighboring phrases within different ranges and predict the text-image semantic similarity in complex scenarios.

[0014] 2. The present invention more comprehensively reflects the text-image semantic similarity by combining local fragment matching, neighboring phrase matching, and multiple non-neighboring phrase matchings.

[0015] 3. The method of the present invention can perform cross-modal retrieval well and has achieved competitive results on multiple databases. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is the flowchart of cross-modal retrieval based on phrase matching combination of the present invention;

[0017] Figure 2 is the performance comparison result of different hyperparameter settings of the present invention on the Flickr30k and MS-COCO datasets;

[0018] Figure 3 is the visualization result of the ablation experiment of the present invention on the Flickr30k dataset;

[0019] Figure 4 is the visualized visual result of text retrieval for a given image query on the Flickr30K dataset of the present invention;

[0020] Figure 5 is the visualized visual result of image retrieval for a given text query on the Flickr30K dataset of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0021] The technical solutions of the present invention will be further described below in conjunction with the accompanying drawings, but are not limited thereto. Any modification or equivalent replacement of the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention shall be covered by the protection scope of the present invention.

[0022] The present invention provides a cross-modal retrieval method based on phrase matching combination. First, a set of grid features is output for each image to represent the image, and then it is mapped to a common semantic space; subsequently, a text backbone network is used to convert the word sequence into a text representation and then mapped to the common semantic space; then, the stacking of different levels of GCNs is introduced to model non-neighboring phrases within different ranges and predict the text-image semantic similarity in complex scenarios; then, by combining local fragment matching, neighboring phrase matching, and multiple non-neighboring phrase matchings, the text-image semantic similarity is more comprehensively reflected; finally, a hinge loss supervised model training with hard negative sample mining is adopted. As Figure 1 shown, the specific steps are as follows:

[0023] Step 1. For the image part in the image-text pair, use the pre-trained ResNeXT network to extract the grid features V' = {v1', …, v j ', …, v n '} of the image I, where v j ' represents the feature vector of the j-th grid after encoding, j = 1, …, n, n represents the number of image grids, and d1 represents the dimension of the image feature vector; Next, apply a linear mapping in the image grid feature space to map the grid features to the common semantic space, obtaining the image feature set V = {v1, …, v j , …, v n}, where is the image grid feature vector in the common semantic space, v j = W v v j ' + b v , W v and b v represent the learnable parameters of the fully connected layer, and d represents the dimension of the common semantic space.

[0024] Step 2. For the text part in the image-text pair, use the text backbone network to convert the word sequence into the text representation U' = {u1', …, u i ', …, u m '}, where: u i ' represents the feature vector of the i-th text word after encoding, i = 1, …, m, m represents the number of words in the sentence, and d2 represents the dimension of the text feature vector; Next, apply a linear mapping in the text feature space to map the grid features to the common semantic space, obtaining the text feature set U = {u1,..., u i ,..., u m}, where is the text word feature vector in the common semantic space, u i = W u u i ' + b u , W u and b u represent the learnable parameters of the fully connected layer.

[0025] Step 3. Calculate the fine-grained similarity matrix S using the image feature set and the text feature set; With the help of the fine-grained similarity matrix, each image grid feature vector v j is represented by the weighted sum of all text features, obtaining the corresponding image grid context feature vector Similarly, each text word feature vector u iIt is represented by the weighted sum of all image grid features to obtain the corresponding text word context feature vector The specific steps are as follows:

[0026] Step 3-1: Calculate the fine-grained similarity matrix S using the image feature set and the text feature set:

[0027]

[0028] where s ij is the normalized dot product of vectors u i and v j representing the cosine similarity between the i-th word and the j-th grid.

[0029] Step 3-2: Use the j-th column of the fine-grained similarity matrix to represent the semantic association between the j-th image grid and all m words, corresponding to the weights of all words. To ensure the rationality and numerical stability of these weights, a preprocessing operation (normalization) is introduced:

[0030]

[0031] where: λ1 represents the hyperparameter of the softmax function; represents the row normalization of S. Construct the image grid context feature set where is defined as follows:

[0032]

[0033] Step 3-3: Use the i-th row of the fine-grained similarity matrix to represent the semantic association between the i-th word and all n image grids, corresponding to the weights of all image grids. To ensure the rationality and numerical stability of these weights, a preprocessing operation (normalization) is introduced:

[0034]

[0035] where: λ2 represents the hyperparameter of the softmax function; represents the column normalization of S. Construct the text word context feature set where is defined as follows:

[0036]

[0037] Step 4: Use the image grid feature set and the image grid context feature set to calculate the visual local segment matching feature V 1 and perform local segment matching Calculate the local segment matching feature U of the text by using the text word feature set and the text word context feature set 1 , and perform local segment matching The specific steps are as follows:

[0038] Step 4-1: Calculate the local segment matching feature of the image by using the image grid feature set and the image grid context feature set

[0039]

[0040] Where: diag(·) represents the operation of returning the diagonal elements of the matrix in vector form; and represent the learnable parameters of the k-th fully connected layer; || represents the concatenation of the local matching outputs from t fully connected layers; The local matching score between the image and text pairs based on the image grid is as follows:

[0041]

[0042] Where and represent the learnable parameters of the fully connected layer.

[0043] Step 4-2: Calculate the local segment matching feature of the text by using the text word feature set and the text word context feature set

[0044]

[0045] Where and represent the learnable parameters of the k-th fully connected layer, and the local matching score between the image and text pairs based on the text word is as follows:

[0046]

[0047] Where and represent the learnable parameters of the fully connected layer.

[0048] Step 5: By using the local segment matching features (V 1 and U 1 ), calculate the neighborhood phrase matching features (V 2 and U 2 ) and the non-neighborhood phrase matching features (V 3 , V 4 , …, V N and U 3 , U 4 , …, U N), respectively perform neighborhood phrase matching ( and ) and non-neighborhood phrase matching ( and ), and the specific steps are as follows:

[0049] Step 5-1: Use the image local fragment matching feature to construct a visual graph The nodes in it are represented as Set the adjacency matrix A by calculating the polar coordinates of each pair of grid centers p (edges). For A p each node in only retains the nearest node as its neighborhood node and the distance between all other nodes is regarded as infinite. and The polar coordinate adjacency condition is defined as:

[0050]

[0051] where: σ ρ and σ θ represent the hyperparameters of A p ; ρ ij and θ ij represent the distance between nodes and in polar coordinates; exp(·) represents the natural exponential function. The adjacency matrix of

[0052] A v = A vs · A p

[0053]

[0054] where "·" represents the Hadamard product, and the image neighborhood phrase matching feature is calculated as follows:

[0055]

[0056] where: is the element in the matrix, is the element in the matrix, represents the identity matrix, represents the learnable parameters of the fully connected layer. The neighborhood matching score between the image text pairs based on the image grid is as follows:

[0057]

[0058] Step Five Two: Utilize text local fragment matching features Construct a text graph The nodes therein are represented as Utilize CoreNLP to analyze the word dependency relationship A', The adjacency matrix of which is defined as:

[0059] A t = A us · A'

[0060]

[0061] Text neighborhood phrase matching features Calculate as follows:

[0062]

[0063] The neighborhood matching scores between image-text pairs based on text words are as follows:

[0064]

[0065] Step Five Three: Refer to Step Five One to calculate the non-neighborhood phrase matching features V 3 , V 4 , …, V N and the non-neighborhood matching scores Refer to Step Five Two to calculate the non-neighborhood phrase matching features U 3 , U 4 , …, U N and the non-neighborhood matching scores where N represents the receptive field range of non-neighborhood matching.

[0066] Step Six: Combine the phrase matching results in Step Five, calculate the similarity s(I, T) between the image and the text, and perform supervision using the hinge loss with hard negative sample mining. The specific steps are as follows:

[0067] Step Six One: Combine the above phrase matching results and calculate the similarity s(I, T) between the image and the text:

[0068]

[0069] Step 62: Use the hinge loss with hard negative sample mining as the objective function to optimize image-text matching. When using text as the query, in each mini-batch, sample the images that match the query and the images that do not match the query separately to form positive pairs and negative pairs. The similarity of the positive pairs should be at least γ higher than the similarity of the negative pairs. Similarly, when using an image as the query, the positive pairs represent the texts that match the given query image in each mini-batch, and the negative pairs represent the texts that do not match the given query image in each mini-batch. The similarity of the negative pairs and the positive pairs also satisfies the above constraints. Therefore, the objective function is defined as follows:

[0070]

[0071] where: γ represents the margin parameter; I' and T' represent negative samples; [·] + represents max[·, 0].

[0072] Experimental results:

[0073] This invention conducts experiments on an Ubuntu server with an NVIDIA RTX 3090 GPU using Python 3.10.12, Pytorch 2.1.0, Pytorch Geometric 2.6.1, and other libraries. This invention uses the pre-trained ResNeXT-101 (32×8d) as the visual embedding function and Bert-base as the text embedding function. The number of image feature vectors is 64, the dimension d1 of the image feature vectors is 2048, the dimension d2 of the text feature vectors is 768, the dimension d of the common semantic space is 1024, and the hyperparameter γ is set to 0.2. This invention trains the CPM for 30 epochs using the Adam optimizer on the training sets of the Flickr30K dataset and the MS-COCO dataset respectively, and validates the CPM on the corresponding validation sets in each epoch. The mini-batch size of this invention is 64, the initial learning rate is set to 0.0002, and the learning rate decays by 10% every 15 epochs. For the evaluation of the test set, this invention uses the recall rate at r (R@r, r = 1, 5, 10) and rSum to measure the performance of the CPM. R@r represents whether, given a query image or text, at least 1 correct text or image is included among the top r retrieved samples. rSum represents the sum of all R@r on the image retrieval text task and the text retrieval image task:

[0074]

[0075] Figure 2Shows the performance comparison results of different λ settings of the present invention on the Flickr30k and MS-COCO datasets. λ determines the relative weights of nodes in the context node representation. A larger λ will filter out most nodes, only retaining a small number of closely related nodes. A smaller λ cannot well distinguish between relevant and irrelevant nodes. Therefore, an appropriate λ is important in the experiment. The present invention conducts experiments with λ = 10, 20 to observe the matching performance in the validation set. Figure 2 (a) shows that in the Flickr30k dataset, CPM produces a better rSum at λ1 = λ2 = 20. Figure 2 (b) shows that in the MS-COCO dataset, CPM yields a better rSum at λ1 = λ2 = 10. The different settings of the parameters may be caused by the different data distributions of the two datasets.

[0076] Figure 3 Shows the visualization results of the ablation experiment of the present invention on the Flickr30k dataset. Given the text query "Alady with blond-hair and glasses works hard to clean up her table.", the present invention shows the local fragment matching (shown in row "a"), neighborhood phrase matching (shown in row "b"), combined local fragment matching and neighborhood phrase matching (shown in row "c") and the top 4 images retrieved on the right side of CPM. The images corresponding to the query are highlighted in the green box. It can be seen that row "a" captures local fragment information such as "lady", "blond-hair", "glasses" and "table", etc. Row "b" captures structured fragment information such as "lady with blond-hair", "lady with glasses" and "lady with table". Row "c" combines local fragments and neighborhood phrases, thus improving the ranking of the GroundTruth. However, they still cannot distinguish it from the leftmost one, resulting in an incorrect TOP-1 retrieval result. However, the CPM of the present invention combines local fragment matching, neighborhood phrase matching and non-neighborhood phrase matching, which can more comprehensively reflect the similarity between samples and retrieve the correct images.

[0077] Figure 4 Shows the visual results of the text retrieval for a given image query on the Flickr30K dataset. Figure 5 Shows the visualization results of the image retrieval for a given text query on the Flickr30K dataset. Here, the truly matching samples are marked in green.

[0078] Table 1 shows the quantitative evaluation results of the present invention and mainstream cross-modal retrieval methods under evaluation criteria such as R@1, R@5, and R@10 in the MS-COCO dataset. For a more comprehensive comparison, the present invention uses two settings (1K and 5K) of the test set on MS-COCO. It can be seen that the present invention achieves the best performance in most metrics for both image retrieval and sentence retrieval tasks, indicating a significant improvement compared to previous methods.

[0079] Comparison of the experimental results of Table 1 with the current mainstream cross-modal retrieval results on the MS-COCO database

[0080]

[0081] Table 2 shows the comparison results of the present invention on the Flickr30K dataset. It can be found that the present invention outperforms other methods under all evaluation metrics, indicating the superiority of the present invention.

[0082] Comparison of the experimental results of Table 2 with the current best cross-modal retrieval results on the Flickr30K database

[0083]

Claims

1. A cross-modal retrieval method based on phrase matching and combination, characterized in that The method includes the following steps: Step 1. For the image part in the image-text pair, use the pre-trained ResNeXT network to extract the grid features V' = {v1', …, v j ', …, v n '} of the image I, where: v j ' represents the feature vector of the j-th grid after encoding, j = 1, …, n, and n represents the number of image grids; apply a linear mapping in the image grid feature space to map the grid features to the common semantic space, obtaining the image feature set V = {v1, …, v j , …, v n}, and v j is the image grid feature vector in the common semantic space; Step 2. For the text part in the image-text pair, use the text backbone network to convert the word sequence into a text representation U' = {u1', …, u i ', …, u m '}, where: u i ' represents the feature vector of the i-th text word after encoding, i = 1, …, m, and m represents the number of words in the sentence; apply a linear mapping in the text feature space to map the grid features to the common semantic space, obtaining a text feature set U = {u1, …, u i , …, u m}, where u i is the text word feature vector in the common semantic space; Step 3: Calculate the fine-grained similarity matrix S using the image feature set and the text feature set; with the fine-grained similarity matrix, each image grid feature vector v j is represented by the weighted sum of all text features to obtain the corresponding image grid context feature vector Similarly, each text word feature vector u i is represented by the weighted sum of all image grid features to obtain the corresponding text word context feature vector Step 4: Calculate the visual local segment matching feature V using the image grid feature set and the image grid context feature set 1 and perform local segment matching Calculate the text local segment matching feature U using the text word feature set and the text word context feature set 1 and perform local segment matching Step 5: Using local fragment matching features, by stacking GCNs, calculate the neighborhood phrase matching feature and the non-neighborhood phrase matching feature respectively, and perform neighborhood phrase matching and non-neighborhood phrase matching respectively; Step 6: Combine the phrase matching results in Step 5, calculate the similarity s(I,T) between the image and the text, and use the hinge loss with hard negative sample mining for supervision.

2. The cross-modal retrieval method based on phrase matching combination according to claim 1, characterized in that In the first step, v j = W v v j '+ b v , where W v and b v represent learnable parameters of the fully connected layer.

3. The cross-modal retrieval method based on phrase matching combination according to claim 1, characterized in that In the second step, u i = W u u i '+ b u , where W u and b u represent the learnable parameters of the fully connected layer.

4. The cross-modal retrieval method based on phrase matching combination according to claim 1, characterized in that The specific steps of Step 3 are as follows: Step 3-1: Calculate the fine-grained similarity matrix S using the image feature set and the text feature set: where s ij is the normalized dot product of vectors u i and v j and represents the cosine similarity between the i-th word and the j-th grid; Step 3-2: Use the j-th column of the fine-grained similarity matrix to represent the semantic association between the j-th image grid and all m words, corresponding to the weights of all words. To ensure the rationality and numerical stability of the weights, introduce a preprocessing operation: Where: λ1 represents the hyperparameter of the softmax function; Indicates that S is normalized row by row; construct the image grid context feature set Where Is defined as follows: Step 3-3: Use the i-th row of the fine-grained similarity matrix to represent the semantic association between the i-th word and all n image grids, corresponding to the weights of all image grids. To ensure the rationality and numerical stability of the weights, introduce a preprocessing operation: Where: λ2 represents the hyperparameter of the softmax function; denotes that S is normalized column-wise; construct the set of text word context features Where is defined as follows:

5. The cross-modal retrieval method based on phrase matching combination according to claim 1, characterized in that The specific steps of Step 4 are as follows: Step 4-1: Calculate the local image segment matching features by using the image grid feature set and the image grid context feature set where: diag(·) represents an operation that returns the diagonal elements of a matrix in vector form; and represents the learnable parameters of the k-th fully connected layer; || represents the concatenation of the local matching outputs from t fully connected layers; the local matching scores between image-text pairs based on the image grid are as follows: Among them and represent the learnable parameters of the fully connected layer; Step 42: Calculate the text local segment matching features by using the text word feature set and the text word context feature set Among them and represent the learnable parameters of the k-th fully connected layer. The local matching scores between image-text pairs based on text words are as follows: Among them and represent the learnable parameters of the fully connected layer.

6. The cross-modal retrieval method based on phrase matching combination according to claim 1, wherein The specific steps of Step 5 are as follows: Step 5.1: Utilize the local image segment matching feature V 1 Construct a visual graph The nodes therein are represented as Set the adjacency matrix A by calculating the polar coordinates of the center of each pair of grids p , for each node in A p only retain the nearest node as its neighborhood node The polar coordinate adjacency conditions of and are defined as:​​​​ where: σ ρ and σ θ represent hyperparameters of A p ; ρ ij and θ ij represent the distance between nodes and in polar coordinates; exp(·) represents the natural exponential function; The adjacency matrix of A v = A us ·A p where "·" represents the Hadamard product, and the image neighborhood phrase matching feature is calculated as follows: Wherein: is an element in the matrix, is an element in the matrix, represents the identity matrix, represents the learnable parameters of the fully connected layer; the neighborhood matching scores between image-text pairs based on the image grid are as follows: Step Five Two: Utilize the text local segment matching feature U 1 Construct a text graph The nodes therein are represented as Utilize CoreNLP to analyze the word dependency relationship A', The adjacency matrix of is defined as: A t = A us · A' Text neighborhood phrase matching feature Calculate as follows: The neighborhood matching scores between image-text pairs based on text words are as follows: Step Five Three: Refer to Step Five One to calculate the non-neighborhood phrase matching feature V based on the image 3 ,V 4 ,…,V N and the non-neighborhood matching score Refer to Step Five Two to calculate the non-neighborhood phrase matching feature U based on the text 3 ,U 4 ,…,U N and the non-neighborhood matching score where N represents the receptive field range of the non-neighborhood matching.

7. The cross-modal retrieval method based on phrase matching combination according to claim 1, wherein The specific steps of Step 6 are as follows: Step 6-1: Combine the phrase matching results and calculate the similarity s(I,T) between the image and the text: Step 6-2: Use the hinge loss with hard negative sample mining as the objective function for optimizing image-text matching. When using the text as the query, in each mini-batch, sample the images that match the query and the images that do not match the query respectively to form positive pairs and negative pairs; when using the image as the query, the positive pair represents the text that matches the given query image in each mini-batch, and the negative pair represents the text that does not match the given query image in each mini-batch; the objective function is defined as follows: Where: γ represents the margin parameter; I' and T' represent negative samples; [·] + represents max[·, 0].

Images