A progressive image-text matching method

Abstract

The application discloses a progressive image-text matching method. In the prior art, the method based on intra-modal interaction encodes image and text features independently, which is very efficient in retrieval, but the performance is not ideal due to lack of sufficient inter-modal interaction. The method based on inter-modal interaction achieves excellent retrieval performance through a large number of inter-modal interactions, but also causes retrieval delay. The intuition of human beings in image-text retrieval is to roughly filter the retrieval samples, eliminate irrelevant samples, then perform more accurate matching on the basis of the candidate set, and finally find the matching item. In this way, the method based on intra-modal and the method based on inter-modal are combined in a progressive manner, and finally the image-text matching method is realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of visual and text matching, and more specifically to a progressive image-text matching method. BACKGROUND

[0002] The image-text matching task is a basic but very critical and challenging task in the field of visual and text, which aims to retrieve the accurate matching of another modality in the given modality query, i.e. image-to-text and text-to-image.

[0003] The existing image-text matching method can be divided into intra-modal interaction and inter-modal interaction according to the different modal interactions. The method based on intra-modal interaction adopts independent image and text encoders to understand and explore the semantic clues in each modality. The existing work usually first converts the image and text into a scene graph, and then uses a graph convolution network to interact within the modal, or directly uses the recently very popular and effective self-attention mechanism to obtain the final representation of the corresponding modality. However, such intra-modal interaction framework lacks inter-modal information interaction, resulting in unsatisfactory final performance. Therefore, the method based on inter-modal is proposed, which usually adopts a deep network with a large number of inter-modal interaction operations, takes image and text as input, and fully explores the interaction between them to achieve better image-text matching.

[0004] The method based on intra-modal interaction can extract the representation of the candidate offline, which makes it very efficient, but due to the lack of rich inter-modal interaction, the performance is not very ideal. While the method based on inter-modal, although the performance is impressive due to a large number of inter-modal interaction operations, but also because of this, greatly increases the computational cost in the reasoning process, ultimately leading to high retrieval delay. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art and provide a progressive image-text matching method. When performing image-text matching on a large amount of data, it will first filter out some very irrelevant samples, and then perform more accurate and detailed matching on the remaining candidate set after filtering.

[0006] In order to achieve the above-mentioned purpose of the application, the progressive image-text matching method of the present application comprises the following steps:

[0007] (1) Establish a retrieval database;

[0008] Select an image I to be retrieved, form a text set containing all the texts matching the image I, and finally take the image I and the corresponding text set as the retrieval database;

[0009] (2) Extracting image text features;

[0010] (2.1) Extracting image features;

[0011] The pre-trained Faster-RCNN object detection model is used to extract image region features from image I, and a confidence score is assigned to each image region. Then, based on the detection confidence score, the top m image regions with the highest scores are selected as image features.

[0012] (2.2) Extracting text features;

[0013] The pre-trained BERT (Transformer-based bidirectional encoder) model is used to extract features from all texts in the retrieved text database, extracting the feature representation of each word in the text, thus forming the text word features;

[0014] (2.3) Mapping of image-text features;

[0015] A fully connected layer is used to map image features and each text feature to the same dimensional space, resulting in the mapped image features V = {v i |v1,v2,…,v m} and the features of each text Among them, v i Represents the features of the i-th image region. Let represent the feature representation of the j-th word in the t-th text, and n represent the number of words in the text;

[0016] (2) Constructing a candidate text library

[0017] (2.1) Set up the model for each single-head self-attention module in the multi-head self-attention layer;

[0018]

[0019] Where Q, K, P represent vectors in the attention operation, the superscript T indicates transpose, and d indicates the dimension of Q and K;

[0020] (2.2) Combine image features V and text features W t The inputs are fed into a multi-head self-attention layer, and the outputs of multiple single-head self-attention modules are concatenated to obtain the complete feature MH(X):

[0021] MH(X)=[H1,H2,…,H l ,…,H h ]

[0022] H l =Attention(XWl Q ,XW l K ,XW l P )

[0023] wherein X represents an input image feature or text feature, H l represents the output of the l-th single-head self-attention module, h represents the number of single-head self-attention modules, W l Q ,W l K ,W l P represents a parameter to be learned;

[0024] (2.3), performing nonlinear mapping on the feature MH(X) by using a feedforward full connection layer to obtain a feature FFN(MH(X)):

[0025] FFN(MH(X)) = σ(MH(X)W1 + b1)W2 + b2

[0026] wherein b1, b2 are constants, W1, W2 are parameters to be learned, and σ(·) is a sigmoid function;

[0027] (2.4), after the image feature and the text feature are processed by steps (2.2)-(2.3), an image feature and a text feature are obtained, and then an overall semantic representation c v of the image feature and an overall semantic representation

[0028]

[0029] wherein is the i-th region feature in is the j-th word feature in

[0030] (2.5), cosine similarity between the overall semantic representation of the image feature and each text feature is calculated:

[0031]

[0032] wherein S t represents the cosine similarity between the overall semantic representation of the image feature and the t-th text feature , and ||·|| represents L2 norm, and the superscript T represents transposition.​​

[0033] (2.6) Sort all similarities from largest to smallest, and select the K texts with the highest similarity to form a candidate text library.

[0034] (3) Exact match;

[0035] (3.1) Capturing location information and phrase-level semantics;

[0036] Find candidate text libraries The original text features corresponding to each candidate text, where the first... The original text features are denoted as

[0037] Spatial location information was added to image feature V using fully connected layers and the location information of image regions, and then a convolutional network was used to add spatial location information to the original text features. Phras-level information was captured, resulting in enhanced image-text features.

[0038] (3.2) Interactive guidance unit;

[0039] (3.2.1) Use global guidance units to guide image features to capture global semantic correspondence information between images and text;

[0040] Enhanced text features Perform average pooling to obtain global text features.

[0041]

[0042] in, For the first text Features of the j-th word;

[0043] Obtain using a fully connected layer Features of each region Global guidance direction d i Then through d i Scaling to guide local visual representation

[0044]

[0045]

[0046] Where ⊙ is the matrix dot product, L2(·) is normalized, and W g b is the parameter to be learned. g It is a constant;

[0047] (3.2.2) Using local guidance units to guide image features to capture local semantic correspondence information between images and text;

[0048] Local guidance information is obtained by aggregating word representations through cross-attention mechanism.

[0049]

[0050]

[0051] Among them, s ij Let λ be the cosine similarity between the i-th region in the image and the j-th word in the text, where λ is a hyperparameter.

[0052] Local guidance information is accessed through a fully connected layer. Projecting yields the corresponding scaling vector γ. i and shift vector β i :

[0053]

[0054]

[0055] Among them, FC γ (·) and FC β (·) represents two fully connected layers;

[0056] Finally, image region guidance is achieved through feedforward fully connected layers and residual connections:

[0057]

[0058] in, represents the image features after guidance in the i-th region, and FFN(·) represents the feedforward fully connected layer.

[0059] (3.2.3) Complete the matching;

[0060] Summing the results of local and global guidance for each region, the sum after guidance for the i-th region is:

[0061] The summed features from each region are then subjected to average pooling to obtain the global image representation V. e ;

[0062]

[0063] Calculate the similarity S among all texts in the candidate text library. e ;

[0064]

[0065] Finally, the text with the largest similarity is selected as the matching result.

[0066] The application object of the present application is achieved as follows:

[0067] The present application proposes a progressive image-text matching method. In existing methods, the method based on intra-modal interaction encodes image and text features independently, which is very efficient in retrieval, but the performance is not ideal due to the lack of sufficient inter-modal interaction. The method based on inter-modal interaction achieves excellent retrieval performance through a large number of inter-modal interactions, but this also causes retrieval delay. The human intuition in image-text retrieval is generally to first roughly filter the retrieval samples and eliminate irrelevant samples, and then perform more accurate matching based on the candidate set to find the matching item. Guided by this human intuition, the method based on intra-modal and inter-modal is combined in a progressive manner to realize the image-text matching method proposed in the present application.

[0068] Meanwhile, the progressive image-text matching method of the present application also has the following beneficial effects:

[0069] (1) The existing image-text matching methods are all single based on intra-modal interaction or single based on inter-modal interaction, which have advantages but also have disadvantages. The former is very efficient, but lacks rich inter-modal information interaction, resulting in unsatisfactory performance. The latter has excellent performance, but has the problem of low efficiency in the reasoning process. The present application combines the rough filter and the accurate matcher in a progressive manner to realize the human intuition-based matching, which is still efficient while maintaining high performance.

[0070] (2) The present application combines the proposed rough filter and accurate matcher in a progressive manner, and the two belong to the method based on intra-modal and the method based on inter-modal respectively. The former only performs intra-modal interaction operation and mainly explores the superficial representation of image and text, while the latter has a large number of inter-modal interaction operations and can explore the subtle representation between image and text. Such characteristics also conform to the human intuition in image-text retrieval. During rough filtering, only shallow features are considered for fast filtering, while during accurate matching, more subtle relationships and features are considered to find the matching item. Therefore, the method proposed in the present application also conforms to human intuition in feature exploration. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1 is a flowchart of the progressive image-text matching method of the present application;

[0072] Figure 2 is a schematic diagram of an interactive guide unit. Detailed Implementation

[0073] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main content of the invention will be omitted here.

[0074] Example

[0075] Figure 1 This is a flowchart of a progressive image-text matching method according to the present invention.

[0076] In this embodiment, as Figure 1 As shown, this invention provides a progressive image-text matching method that can perform image-text matching by searching for text in the image or searching for images in the text. The matching principle and process are the same. Below, we will analyze and explain using image-text matching as an example, specifically including the following steps:

[0077] S1. Establish a retrieval database;

[0078] Select an image I to be searched, and form a text set by combining all texts that contain matching texts of image I. Finally, use image I and the corresponding text set as the search database.

[0079] S2. Extract image text features;

[0080] S2.1 Extracting image features;

[0081] The pre-trained Faster-RCNN object detection model is used to extract image region features from image I, and a confidence score is assigned to each image region. Then, based on the detection confidence score, the top m = 36 image regions with the highest scores are selected as image features.

[0082] S2.2 Extracting text features;

[0083] The pre-trained BERT (Transformer-based bidirectional encoder) model is used to extract features from all texts in the retrieved text database, extracting the feature representation of each word in the text, thus forming the text word features;

[0084] S2.3 Mapping of image and text features;

[0085] A fully connected layer is used to map image features and each text feature to the same dimensional space with a dimension of 2048, resulting in the mapped image features V = {v i |v1,v2,…,v m} and the features of each text wherein v i represents the feature of the i-th image region, represents the feature representation of the j-th word in the t-th text, and n represents the number of words in the text;

[0086] S2, constructing a candidate text library

[0087] S2.1, in order to realize the human intuition-like rapid filtering of irrelevant samples, the embodiment adopts a coarse filter module of a self-attention mechanism to perform screening and filtering, and sets the model of each single-head self-attention module in the multi-head self-attention layer to be:

[0088]

[0089] wherein Q, K, and P represent vectors in the attention operation, the superscript T represents transposition, and d represents the dimension size of Q and K;

[0090] S2.2, inputting the image feature V and the text feature W t to the multi-head self-attention layer respectively, and obtaining the complete feature MH(X) by splicing the outputs of the multiple single-head self-attention modules:

[0091] MH(X) = [H1, H2, …, H l ,…, H h ]

[0092] H l = Attention(XW l Q ,XW l K ,XW l P )

[0093] wherein X represents the input image feature or text feature, H l represents the output of the l-th single-head self-attention module, h represents the number of single-head self-attention modules, and is specifically 16, W l Q ,W l K ,W l P represent parameters to be learned;

[0094] S2.3, performing nonlinear mapping on the feature MH(X) by using a feedforward full connection layer to obtain the feature FFN(MH(X)):

[0095] FFN(MH(X)) = σ(MH(X)W1 + b1)W2 + b2

[0096] where b1, b2 are constants, W1, W2 are parameters to be learned, and σ(·) is a sigmoid function.

[0097] S2.4, after the image features and the text features are processed by steps S2.2-S2.3, the image features and the text features are obtained. v Then, the overall semantic representation c of the image features and the overall semantic representation of the text features are obtained through an average pooling operation.

[0098]

[0099] wherein, is the i-th region feature in the image features, is the j-th word feature in the text features.

[0100] S2.5, the cosine similarity between the overall semantic representation of the image features and each text feature is calculated.

[0101]

[0102] wherein, S t represents the cosine similarity between the overall semantic representation of the image features and the t-th text feature , and ||·|| represents the L2 norm, and the superscript T represents transposition.

[0103] S2.6, all the similarities are arranged in descending order, and the top K similarities are selected to form a candidate text library.

[0104] S3, accurate matching.

[0105] After the rough filtering in step S2, a guide unit with rich inter-modal interaction is used to perform further accurate matching similar to human intuition on the candidate text library, and the specific process is as follows:

[0106] S3.1, capture of position information and phrase-level semantics.

[0107] Find the original text features corresponding to each candidate text in the candidate text library , wherein the i-th original text feature is denoted as

[0108] Spatial location information was added to image feature V using fully connected layers and the location information of image regions, and then a convolutional network was used to add spatial location information to the original text features. Phras-level information was captured, resulting in enhanced image-text features.

[0109] S3.2, Interactive Guide Unit;

[0110] Image and text features are fed into interactive guidance units. Two types of interactive guidance units are used here: local guidance units and global guidance units. The two guidance units are arranged in parallel and then stacked to perform in-depth information exploration and semantic alignment. The specific guidance process is as follows:

[0111] S3.2.1, such as Figure 2 As shown in (a), the global guidance unit is used to guide image features to capture global semantic correspondence information between images and text;

[0112] Enhanced text features Perform average pooling to obtain global text features.

[0113]

[0114] in, For the first text Features of the j-th word;

[0115] Obtain using a fully connected layer Features of each region Global guidance direction d i Then through d i Scaling to guide local visual representation

[0116]

[0117]

[0118] Where ⊙ is the matrix dot product, L2(·) is normalized, and W g b is the parameter to be learned. g It is a constant;

[0119] S3.2.2, such as Figure 2 As shown in (b), local guidance units are used to guide image features to capture local semantic correspondence information between images and text;

[0120] Local guidance information is obtained by aggregating word representations through cross-attention mechanism.

[0121]

[0122]

[0123] Among them, s ij Let λ be the cosine similarity between the i-th region in the image and the j-th word in the text, where λ is a hyperparameter.

[0124] Local guidance information is accessed through a fully connected layer. Projecting yields the corresponding scaling vector γ. i and shift vector β i :

[0125]

[0126]

[0127] Among them, FC γ (·) and FC β (·) represents two fully connected layers;

[0128] Finally, image region guidance is achieved through feedforward fully connected layers and residual connections:

[0129]

[0130] in, Let represent the image features after guidance for the i-th region, and FFN(·) represent the feedforward fully connected layer;

[0131] S3.2.3 Complete the matching;

[0132] like Figure 1 As shown in the bottom right corner, the input of each layer of the stacked interactive guidance module is the sum of the outputs of the local and global guidance units of the previous layer. Therefore, by summing the results of local and global guidance for each region, the summation after guiding the i-th region is as follows:

[0133] The summed features from each region are then subjected to average pooling to obtain the global image representation V. e ;

[0134]

[0135] Calculate the similarity S among all texts in the candidate text library. e ;

[0136]

[0137] Finally, the text with the highest similarity is selected as the matching result.

[0138] While the foregoing specific embodiments of the application have been described in some detail to provide a clear understanding thereof, it will be apparent to those of ordinary skill in the art that numerous modifications can be made to the specific embodiments described without departing from the spirit and scope of the application defined by the appended claims.

Claims

1. A progressive image-text matching method, characterized in that, Includes the following steps: (1) Establish a retrieval database; Select an image to be searched , will contain images All text matching the text is combined into a text set, and finally the image is... and the corresponding text set as the retrieval database; (2) Extracting image text features; (2.1) Extracting image features; The image is processed using a pre-trained Faster-RCNN object detection model. Image region features are extracted, and a confidence score is assigned to each image region. Then, based on the detection confidence score, the regions with the highest scores are selected. Each image region is used as an image feature; (2.2) Extracting text features; The pre-trained BERT model is used to extract features from all texts in the retrieved text database, extracting the feature representation of each word in the text to form the text word features; (2.3) Mapping of image-text features; A fully connected layer is used to map image features and each text feature to the same dimensional space, resulting in mapped image features. With the features of each text ,in, Indicates the first Features of an image region This indicates the t-th text. Feature representation of each word Indicates the number of words in the text; (3) Constructing a candidate text library ; (3.1) Set up the model for each single-head self-attention module in the multi-head self-attention layer; ； in, The vector in the attention operation is indicated by the superscript. Indicates transpose. express Dimension size; (3.2) Image features and text features The inputs are fed into a multi-head self-attention layer, and the complete features are obtained by concatenating the outputs of multiple single-head self-attention modules. : ； ； Where X represents the input image features or text features. Indicates the first The output of a single-head self-attention module Indicates the number of single-head self-attention modules. , , Indicates the parameter to be learned; (3.3) Utilizing a feedforward fully connected layer for feature processing Perform nonlinear mapping to obtain features : ； in, It is a constant. The parameters to be learned It is the sigmoid function; (3.4) After processing in steps (2.2)-(2.3), image features and text features are obtained. Text features Then, average pooling is used to obtain the overall semantic representation of the image features. Overall semantic representation of text features ; ； in, for The Middle Regional characteristics, for The Middle Each word's characteristics; (3.5) Calculate image features With each text feature Cosine similarity of the overall semantic representation: ； in, Representing image features With the Text features The cosine similarity of the overall semantic representation. This indicates the L2 norm, with superscript indicating the value. Indicates transpose; (3.6) Sort all similarities from largest to smallest, and select the K texts with the highest similarity to form a candidate text library. ; (4) Exact match; (4.1) Capturing location information and phrase-level semantics; Find candidate text libraries The original text features corresponding to each candidate text, where the first... The original text features are denoted as ; Image features are generated using fully connected layers and the location information of image regions. Spatial location information was added, and then a convolutional network was used to generate features for the original text. Phras-level information was captured, resulting in enhanced image-text features. , , ; (4.2) Interactive guidance unit; (4.2.1) Using global guidance units to guide image features to capture global semantic correspondence information between images and text; Enhanced text features Perform average pooling to obtain global text features. ; ； in, For the first text The Middle Each word's characteristics; Obtain using a fully connected layer Features of each region Global guidance direction Then through Scaling to guide local visual representation ; ； ； in, For matrix dot product, Normalization The parameters to be learned It is a constant; (4.2.2) Using local guidance units to guide image features to capture local semantic correspondence information between images and text; Local guidance information is obtained by aggregating word representations through cross-attention mechanism. : ； ； in, Let be the cosine similarity between the i-th region in the image and the j-th word in the text. For hyperparameters; Local guidance information is accessed through a fully connected layer. Project the vector to obtain the corresponding scaling vector. and shift vector : ； in, and It consists of two fully connected layers; Finally, image region guidance is achieved through feedforward fully connected layers and residual connections: ； in, This represents the image features after guidance for the i-th region. Indicates a feedforward fully connected layer; (4.2.3) Complete the matching; Summing the results of local and global guidance for each region, the sum after guidance for the i-th region is: ; Average pooling is performed on the summed features of each region to obtain the global image representation. ; ； Calculate the similarity of all texts in the candidate text library. ; ； Finally, the text with the highest similarity is selected as the matching result.

Images