An image-text matching method based on similarity random mixed reasoning

Abstract

The application relates to an image-text matching method based on similarity random mixed reasoning, which comprises the following steps: calculating an image-to-text local similarity representation according to given pairs of images and texts; inputting the local similarity representation into a similarity random mixed reasoning network to obtain inferred local similarity information; performing a nonlinear mapping operation to obtain a global similarity representation; the similarity random mixed reasoning network comprises a grouping mapping module, a random mixed module, a multi-head attention network and a feedforward neural network which are connected in sequence; after the local similarity representation block is subjected to a grouping mapping operation, the local similarity representation block is spliced, the spliced result is subjected to a random mixing operation, the result after the random mixing operation is input into the multi-head attention network to perform diversified relationship reasoning, and finally the result is input into the feedforward neural network to obtain the inferred local similarity information. Compared with the prior art, the application has the advantages of comprehensive relationship reasoning, strong feature learning capability and diversified interaction.

Description

Technical Field

[0001] This invention relates to the field of image-text matching technology, and in particular to an image-text matching method based on similarity-based random hybrid reasoning. Background Technology

[0002] Image-text matching aims to measure the similarity between images and text and is widely used in cross-media information interaction tasks, such as cross-media retrieval, image captioning, and text-image synthesis. Due to the inherent information exchange capabilities of graph reasoning mechanisms and Transformers, current research focuses on leveraging them to construct cross-media semantic associations. Despite significant progress, the following problems remain:

[0003] (1) Insufficient reasoning space. Graph reasoning can only establish a single reasoning space to facilitate information exchange, while ignoring the exploration of diverse relationships. The multi-head attention mechanism inherent in Transformer can be an alternative to solve this problem. However, the features involved in multi-head attention in Transformer come from complete mapping, so all reasoning spaces constructed by multi-head attention are extremely similar.

[0004] (2) Insufficient feature learning capability. Graph convolution is implemented through pointwise operations, resulting in insufficient feature learning capability. Although the feedforward neural network in Transformer has powerful feature learning capabilities, its performance is limited because its input comes from interaction features that have not been fully reasoned.

[0005] (3) The Transformer itself has a weak global similarity representation capability. The traditional Transformer modeling global representation is based directly on the original image-text pairs, ignoring the importance of multi-level cross-media interaction. Therefore, without pre-training on a large dataset, the insurmountable cross-media semantic gap problem cannot be solved. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology and provide an image-text matching method based on similarity-based random hybrid reasoning. This invention has the advantages of comprehensive reasoning relationships, strong feature learning ability, and diverse interactions.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] This invention provides an image-text matching method based on similarity-based random hybrid inference. The method calculates the local similarity representation from image to text based on a given pair of images and text, inputs the local similarity representation into a similarity-based random hybrid inference network to obtain inferred local similarity information, and performs a nonlinear mapping operation on the inferred local similarity information to obtain a global similarity representation that can measure the similarity between a given image and text.

[0009] The similarity random hybrid reasoning network includes a grouping mapping module, a random hybridization module, a multi-head attention network, and a feedforward neural network connected in sequence. The local similarity representation is grouped and mapped by the grouping mapping module, then concatenated. The concatenated result is input into the random hybridization module to perform a random hybridization operation. The randomly hybridized result is input into the multi-head attention network for diversified relationship reasoning, and finally input into the feedforward neural network to obtain the reasoned local similarity information.

[0010] Preferably, the specific process of performing grouping mapping operation on the image-to-text local similarity representation F is as follows:

[0011] Determine the learnable location embedding P, and add the learnable location embedding P to the local similarity representation F to obtain a location-aware local similarity representation, specifically F←F+P; decompose the location-aware F into h groups along the channel dimension, resulting in F={F1,F2,...,F h Finally, each set of features is projected into a unique feature space through two mapping operations to obtain the mapped features, as shown in the following formula:

[0012]

[0013] in, and For learnable parameters, and For the corresponding bias term, ReLU is the activation function. is the mapped feature, and k is the grouping index.

[0014] Preferably, the process of performing a random mixing operation on the mapped features is as follows:

[0015] Mapped features The splicing operation is performed along the channel dimension, and the features are shuffled by channel shuffling. The specific formula is as follows:

[0016]

[0017] In the formula, [*] represents the splicing operation, F shuffle These are characteristics of the mixture after washing.

[0018] Preferably, the process of performing diversified relational reasoning on the randomly mixed results is as follows:

[0019] The characteristic F after mixing shuffle The features are then divided into h groups along the channel dimension; each h group of features is input into its respective self-attention network for diverse relational reasoning; the outputs of each self-attention network are then concatenated along the channel dimension into a single whole, accompanied by a residual connection operation, as shown in the following formula:

[0020]

[0021] In the formula, [*] represents the concatenation operation, SA(*) represents the self-attention operation, and F sa This is a characteristic of reasoning through diverse relationships.

[0022] Preferably, the process of performing a linear transformation operation on the features that have undergone diversified relational reasoning through a feedforward neural network is as follows:

[0023] The feedforward neural network comprises two consecutive linear transformation operations, which process the feature F after diverse relational reasoning. sa The specific steps are as follows:

[0024] F ffn =F sa +σ(F sa W 3 +b 3 W 4 +b 4

[0025] Where σ(*) represents the nonlinear activation function, W 3 and W 4 For a learnable matrix, b 3 and b 4 These are the corresponding bias terms, F ffn This refers to the local similarity information after inference.

[0026] Preferably, the loss function L during training of the similarity-based random hybrid inference network is:

[0027] L=[α-S r (v,t)+S r (v,t - )] + +[α-S r (v,t)+S r (v - ,t)] +

[0028] Where α is the distance parameter, [x] +=max(x,0), S r This represents the inner product calculation, where v and t are the paired image and text, respectively. - and t - These are the hardest negative samples corresponding to v and t, respectively.

[0029] Preferably, the process of calculating the image-to-text local similarity representation based on the given image-text pairs is as follows:

[0030] Given a pair of images and text, obtain the feature codes corresponding to the images and text respectively, including local feature codes and global feature codes, and calculate the local similarity representation from image to text based on the feature codes of the images and text.

[0031] Preferably, the process of generating image feature codes is as follows:

[0032] A pre-trained object detection network is used to extract K salient regions from an image. Each salient region is mapped to a d-dimensional vector, and this vector is used as the visual local region feature encoding V = {v1, v2, ..., v...}. K}, calculate the global feature code of the image based on the visual local region feature code V. in i is an index, i∈(1,K) The average value of the feature encoding of a local visual region. This indicates an element-wise multiplication operation.

[0033] Preferably, the process of generating text feature codes is as follows:

[0034] The text is decomposed into L word tokens, and word feature codes T = {t1, t2, ..., t3} are obtained based on the L word tokens. L} Calculate the global feature code of the text based on the word feature code T. in, j is the index, j∈(1,L), The average value of the word feature encoding. This indicates an element-wise multiplication operation.

[0035] Preferably, the process of calculating the local similarity representation set from image to text is as follows:

[0036] Calculate global feature encoding of an image and text global feature encoding similarity between Calculate the feature encoding t of the j-th word j Visual features of interest Where, α ij Representing visual region features v i and text word features tj Attention weights between them; calculation of visual features and word feature encoding t j The similarity between them is used to obtain a local similarity representation from region to word. splicing s r2w and s glo To obtain the local similarity representation set F = [s r2w ,s glo ].

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] (1) This invention proposes group mapping to replace the full mapping in the traditional Transformer, solving the problem of insufficient inference space that has not been addressed in graph networks and Transformers. Furthermore, it proposes channel shuffling operation to promote mutual communication between multiple inference spaces to avoid the "edge effect" problem.

[0039] (2) This invention inherits the feedforward neural network inherent in Transformer, and because the feedforward neural network receives relational features that have been fully reasoned, the feedforward neural network performs more significantly, thereby solving the problem of insufficient feature learning ability in graph convolutional networks and Transformer.

[0040] (3) This invention is the first to use Transformer to perform relational reasoning on the local similarity information of region-word pairs. It takes advantage of the importance of multi-level cross-media interaction and aligns cross-media semantic features more densely. Therefore, it can still solve the problem of weak global similarity representation ability in Transformer without the participation of large datasets. Attached Figure Description

[0041] Figure 1 This is a flowchart of an image-text matching method based on similarity-based random hybrid reasoning, provided in this embodiment.

[0042] Figure 2 For the diverse reasoning of existing technologies and Figure 1 A comparison diagram of the workings of the embodiments shown.

[0043] Figure 3 for Figure 1 The illustrated embodiment is shown in a visual representation. Detailed Implementation

[0044] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0045] refer to Figure 1 As shown, this embodiment provides an image-text matching method based on similarity-based random hybrid reasoning, including the following steps:

[0046] Step 1: Obtain paired images and text (v,t), characterize the feature codes of the images and text respectively, and calculate the local similarity representation from image to text based on the feature codes of the images and text;

[0047] Step 11: Generate Image Feature Encoding: Extract K salient regions from the image using a pre-trained object detection network. Map each salient region to a d-dimensional vector using a linear transformer, and use this vector as the visual local region feature encoding V = {v1, v2, ..., v...}. K}, calculate the global feature code of the image based on the visual local region feature code V. in i is the index of the salient region, i∈(1,K). The average value of the feature encoding of a local visual region. This indicates an element-wise multiplication operation.

[0048] As an optional implementation, the object detection network is the Faster-RCNN network.

[0049] Step 12: Generate text feature encoding: Decompose the text into L word tokens using language segmentation methods, and sequentially feed the L word tokens into a bidirectional gated recurrent network (BiGRU) to obtain word feature encodings T = {t1, t2, ..., t...}. L} Calculate the global feature code of the text based on the word feature code T. in, j is the index of the word count, j∈(1,L), The average value of the word feature encoding. This indicates an element-wise multiplication operation.

[0050] Step 13: Calculate the local similarity representation set from image to text: Calculate the global feature encoding of the image. and text global feature encoding similarity between The feature encoding t of the j-th word is calculated using a mutual attention mechanism. j Visual features of interest Where, α ij Representing visual region features v i and text word features t j Attention weights between them; calculation of visual features and word feature encoding tj The similarity between them is used to obtain a region-to-word local similarity representation. splicing s r2w and s glo To obtain the local similarity representation set F = [s r2w ,s glo The specific method for calculating the similarity equation S(*1,*2) is as follows: |*| 2 This represents squaring pixel by pixel, ||*||2 represents L2 regularization, and W is the learnable matrix.

[0051] Step 2: Input the image-to-text local similarity representation F into the similarity random hybrid inference network to obtain the inferred local similarity information.

[0052] refer to Figure 1 As shown, the similarity random hybrid reasoning network includes a grouping mapping module, a random hybridization module, a multi-head attention network, and a feedforward neural network connected in sequence. The image-to-text local similarity representation F is grouped and mapped by the grouping mapping module, laying the foundation for the construction of a diverse reasoning space. Then, the results of the grouping mapping operation are concatenated and input into the random hybridization module to perform a random hybridization operation, promoting the exchange of multiple sets of information and avoiding the influence of edge effects. Next, the randomly hybridized results are fed into the multi-head attention network for diverse relationship reasoning, highlighting important similarity information and suppressing useless similarity features. Finally, the feedforward neural network is used to further exchange information on the reasoned results.

[0053] Step 21: The specific process of performing grouping mapping operation from image to text local similarity representation F is as follows:

[0054] First, determine the learnable location embedding P. Then, add the learnable location embedding P to the local similarity representation F to obtain a location-aware local similarity representation, specifically, F←F+P. Next, decompose the location-aware F into h groups along the channel dimension. The result of this step is F={F1,F2,...,F...} h Finally, each set of features is projected into a unique feature space through two mapping operations to obtain the mapped features, as shown in the following formula:

[0055]

[0056] in, and For learnable parameters, and For the corresponding bias term, ReLU is the activation function. is the mapped feature, and k is the grouping index.

[0057] Step 22: Random Mixing Operation: Mapping the Features The splicing operation is performed along the channel dimension, and the features are shuffled by channel shuffling. The specific formula is as follows:

[0058]

[0059] In the formula, [*] represents the splicing operation, F shuffle These are characteristics of the mixture after washing.

[0060] Step 23: Perform diversified relational reasoning: Mix the shuffled features F shuffle The features are then divided into h groups along the channel dimension; each h group's features are input into its respective self-attention network for diverse relational reasoning; the outputs of each self-attention network are then concatenated along the channel dimension to form a whole, accompanied by a residual connection operation to facilitate information flow. The specific formula is as follows:

[0061]

[0062] In the formula, [*] represents the concatenation operation, SA(*) represents the self-attention operation, and F sa These are the initial features after diverse relational reasoning.

[0063] Step 24: A feedforward neural network is used to perform a linear transformation operation on the initial features obtained from diverse relational reasoning, enhancing the representational power of these features and obtaining enhanced local similarity information for reasoning. The feedforward neural network contains two consecutive linear transformation operations, where the first linear transformation is followed by a non-linear activation function, F... sa The specific steps are as follows:

[0064] F ffn =F sa +σ(F sa W 3 +b 3 W 4 +b 4

[0065] Where σ(*) represents a nonlinear activation function, specifically the Sigmoid function in this paper; W 3 and W 4 For a learnable matrix, b 3 and b 4 For the corresponding bias term, F ffn Local similarity information to enhance reasoning.

[0066] Step 3: Perform a nonlinear mapping operation on the reasoning-enhanced local similarity information to obtain a global similarity representation that can measure the similarity between a given image and text.

[0067] As an optional implementation, the inferred local similarity information F ffn By inputting the data into a fully connected layer with non-linear activation, a global similarity representation that measures the similarity between a given image and text is obtained, as shown in the following formula:

[0068] F glo =σ(F ffn W 5 +b 5 )

[0069] In the formula, σ(*) is the nonlinear activation function Sigmoid; W 5 Let b represent the learnable matrix. 5 For the corresponding bias term, F glo This represents the global similarity.

[0070] As an optional implementation, the loss function L during training of the similarity-based random hybrid inference network is expressed as:

[0071] L=[α-S r (v,t)+S r (v,t - )] + +[α-S r (v,t)+S r (v - ,t)] +

[0072] Where α represents the distance parameter, [x] + =max(x,0), S r This represents the inner product calculation, where v and t represent the paired image and text, respectively. - and t - Let v and t represent the hardest negative samples, respectively.

[0073] This method was experimentally validated on two public datasets (MSCOCO and Flickr30K). The MSCOCO dataset contains 123,287 images, each corresponding to 5 text descriptions. 113,287 images were selected as the training set, 5,000 as the validation set, and 5,000 as the test set. The final result is the average accuracy of the five 1,000-image test sets (MSCOCO 1K), or the accuracy on the entire 5,000-image test set (MSCOCO 5K). The Flickr30K dataset contains 31,783 images, of which 29,783 were used for training, 1,000 for validation, and 1,000 for the test set. The evaluation metric used in the experiments is R@K, which represents the probability that the first K search results in a batch contain the correct item.

[0074] To evaluate the effectiveness of this invention on different datasets, experimental results on MSCOCO 1K and Flickr30K are shown in Table 1, where Ours represents the similarity-based randomized hybrid inference network. It can be seen that the similarity-based randomized hybrid inference network outperforms all state-of-the-art methods on the R@1 metric. To evaluate the effectiveness of this invention on the MSCOCO 5K dataset, experimental results are shown in Table 2. The results show that the similarity-based randomized hybrid inference network still outperforms all state-of-the-art methods on the MSCOCO 5K dataset, demonstrating the strong robustness of similarity-based randomized hybrid inference.

[0075] Table 1 compares experimental results on MSCOCO 1K and Flickr 30K, with bold indicating best performance.

[0076]

[0077]

[0078] Table 2 compares experimental results on MSCOCO 5K, with bold indicating best results.

[0079]

[0080] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. An image-text matching method based on similarity-based random hybrid reasoning, characterized in that, Given a pair of images and text, calculate the image-to-text local similarity representation, input the local similarity representation into a similarity random hybrid inference network to obtain the inferred local similarity information, and perform a nonlinear mapping operation on the inferred local similarity information to obtain a global similarity representation that can measure the similarity between a given image and text. The similarity random hybrid reasoning network includes a grouping mapping module, a random hybridization module, a multi-head attention network, and a feedforward neural network connected in sequence. The local similarity representation is grouped and mapped by the grouping mapping module, then concatenated. The concatenated result is input into the random hybridization module to perform a random hybridization operation. The random hybridized result is input into the multi-head attention network for diversified relationship reasoning, and finally input into the feedforward neural network to obtain the reasoned local similarity information. Representing local similarity between images and text F The specific process of performing group mapping operation is as follows: Determine learnable location embeddings Embed learnable locations Representation of local similarity F The summation yields a location-aware local similarity representation, specifically formulated as follows: ; will have location awareness F Decomposed along the channel dimension h Group, get Finally, each set of features is projected into a unique feature space through two mapping operations to obtain the mapped features, as shown in the following formula: in, and For learnable parameters, and For the corresponding bias term, RELU For activation function, For the mapped features, k Grouped index; The process of performing random mixing on the mapped features is as follows: Mapped features The splicing operation is performed along the channel dimension, and the features are shuffled by channel shuffling. The specific formula is as follows: In the formula, This indicates a splicing operation. These are characteristics of the mixture after washing.

2. The image-text matching method based on similarity-based random hybrid reasoning according to claim 1, characterized in that, The process of performing diversified relational reasoning on the results of random mixing is as follows: Characteristics after mixing Divide again along the channel dimension h Group; will h Each set of features is input into its own self-attention network for diverse relational reasoning. The outputs of each self-attention network are then concatenated along the channel dimension into a single whole, accompanied by a residual connection operation, as shown in the following formula: In the formula, This indicates a splicing operation. This indicates a self-attention operation. This is a characteristic of reasoning through diverse relationships.

3. The image-text matching method based on similarity-based random hybrid reasoning according to claim 2, characterized in that, The process of performing linear transformation on features that have undergone diverse relational reasoning using a feedforward neural network is as follows: The feedforward neural network contains two consecutive linear transformation operations to process features that have undergone diverse relational reasoning. The specific steps are as follows: in, Represents a non-linear activation function. and For learnable matrices, and These are the corresponding bias terms. This refers to the local similarity information after inference.

4. The image-text matching method based on similarity-based random hybrid reasoning according to claim 1, characterized in that, The loss function of the similarity-based random hybrid inference network during training. for: in, For distance parameters, , Indicates inner product calculation. and The images and text are paired respectively. and They are respectively and The corresponding most difficult negative sample.

5. The image-text matching method based on similarity-based random hybrid reasoning according to claim 1, characterized in that, The process of calculating the local image-to-text similarity representation based on a given pair of images and text is as follows: Given a pair of images and text, obtain the feature codes corresponding to the images and text respectively, including local feature codes and global feature codes, and calculate the local similarity representation from image to text based on the feature codes of the images and text.

6. The image-text matching method based on similarity-based random hybrid reasoning according to claim 5, characterized in that, The process of generating image feature codes is as follows: Extracting images using a pre-trained object detection network There are salient regions, and each salient region is mapped to... A 3D vector is used to encode the features of a local visual region. Encoding based on visual local region features Calculate global feature encoding of an image ,in , i For indexing, , The average value of the feature encoding of a local visual region. This indicates an element-wise multiplication operation.

7. The image-text matching method based on similarity-based random hybrid reasoning according to claim 6, characterized in that, The process of generating text feature codes is as follows: Decompose the text into Each word is marked, according to Each word tag obtains a word feature encoding. Encoding based on word features Calculate the global feature encoding of the text ,in, , j For indexing, , The average value of the word feature encoding. This indicates an element-wise multiplication operation.

8. The image-text matching method based on similarity-based random hybrid reasoning according to claim 7, characterized in that, The process of calculating the local similarity representation set from image to text is as follows: Calculate global feature encoding of an image and text global feature encoding similarity between ; Calculate the first Word feature encoding Visual features of interest ,in, Representing visual region features and text word features Attention weights between them; calculation of visual features and word feature encoding The similarity between them is used to obtain a local similarity representation from region to word. splicing and To obtain a local similarity representation set .

Images