A cross-modal retrieval method based on relation enhancement strategy gradient and parallel weight control

Abstract

This invention discloses a cross-modal retrieval method based on relation enhancement policy gradient and parallel weight control. The method is as follows: 1. Output a set of salient region features to represent each image; 2. Encode each token into a word vector using word encoding, converting the word sequence into a text representation; 3. Use policy gradient and parallel weight control to guide the weights of all image region features in the final global image features, obtaining global image features; 4. Use policy gradient and parallel weight control to guide the weights of all text word features in the final global text features, obtaining global text features; 5. Perform similarity matching using the global image features and global text features, supervised by hinge loss with hard negative sample mining, and supervise the weights of image region features and text word features using the optimization objective of discrete continuous policy gradient. This invention can perform cross-modal retrieval well.

Description

Technical Field

[0001] This invention relates to a cross-modal retrieval method, specifically a cross-modal retrieval method based on relation enhancement strategy gradient and parallel weight control. Background Technology

[0002] With the rapid development of multimedia, the internet is teeming with information, including images, text, video, and audio. Manually extracting useful information across different modalities from this massive dataset is becoming increasingly difficult. Naturally, we need a powerful method to help us obtain the text, images, or videos we need. Cross-modal retrieval uses one modality of data as a query to retrieve relevant data in another modality. For example, we can use text to retrieve images of interest (as we do on Google Image Search), or use images to retrieve corresponding text. Of course, modalities are not limited to images and text; other modalities such as speech, physical signals, and video can also be components of cross-modal retrieval. Many recent cross-modal retrieval methods use pre-trained object detectors to process image features. Images typically contain a wealth of visual information, but not all target regions are mentioned in the image description, so it's necessary to control the degree to which each target region feature contributes to the final global feature vector of the image. Previous work used a direct averaging approach for each region feature, treating all target regions equally, but this proved ineffective. Some works have proposed using attention mechanisms to determine the importance of each region feature to the final global feature vector of the image, achieving some progress. However, these methods generally lack effective supervision of the attention mechanism, which limits the improvement of attention mechanisms in cross-modal retrieval tasks. Similar to image branching, text sentences typically contain nouns, verbs, adjectives, adverbs, prepositions, etc., and not all words contribute to the final global feature vector of the text. Therefore, it is necessary to control the degree of participation of each word's features in the final global feature vector of the text. Summary of the Invention

[0003] To improve cross-modal retrieval, this invention provides a cross-modal retrieval method based on relation enhancement strategy gradient and parallel weight control.

[0004] The objective of this invention is achieved through the following technical solution:

[0005] A cross-modal retrieval method based on relation enhancement strategy gradient and parallel weight control includes the following steps:

[0006] Step 1: For the image portion of the image-text pair, use BUA (Bottom Up Attention) as the base image network for the image branch. Select the image in the image-text pair as the input to the image branch, and output a set of salient region features V = {v1, ..., v1} for each image I.i ,…,v n Let} represent the image, where: v i d represents the feature vector of the i-th salient region being encoded, i = 1, ..., n, where n represents the number of salient regions in the image and d represents the dimension of the feature vector;

[0007] Step 2: For the text portion of the image-text pair, use word encoding to encode each token into a word vector. Then, use a bidirectional GRU as the base text network for the text branch to transform the word sequence into a text representation U = {u1, ..., u...}. i ,…,u m},in: u i Let i represent the feature vector of the i-th text word being encoded, where i = 1, ..., m, and m represents the number of words in the sentence;

[0008] Step 3: Model the relationship between arbitrary image region features and all image region features using a self-attention mechanism, and use policy gradient and parallel weight control to guide the weight of all image region features in the final global image features, thus obtaining the global image features v. gl ;

[0009] Step 4: Model the relationship between arbitrary text word features and all text word features using a self-attention mechanism, and use policy gradient and parallel weight control to guide the weight of all text word features in the final global text feature, thus obtaining the global text feature u. gl ;

[0010] Step 5: Perform similarity matching using global image features and global text features, and supervise the process with hinge loss with hard negative sample mining. Use the optimization objective of discrete continuous policy gradient to supervise the weights of image region features and text word features.

[0011] Compared with the prior art, the present invention has the following advantages:

[0012] 1. This invention utilizes a self-attention mechanism to model the relationship between any local feature within a modality and all local features, thereby more accurately using discrete and continuous policy gradients to estimate the weight of the local feature in the final global feature.

[0013] 2. The parallel weight control proposed in this invention significantly improves the training and inference efficiency of cross-modal retrieval models.

[0014] 3. The method of the present invention can perform cross-modal retrieval well, achieve competitive results on multiple databases, and has high training and inference efficiency. Attached Figure Description

[0015] Figure 1 This is a flowchart of the cross-modal retrieval process based on relation enhancement strategy gradient and parallel weight control in this invention;

[0016] Figure 2 The training process of this invention on Flickr30k for 30 epochs is shown below. DCPG represents the baseline method of this invention, and PGRR represents the method proposed in this invention. In (a), the x-axis and y-axis represent Step and Loss, respectively, indicating that the method of this invention converges to a lower Loss under the same epoch conditions. In (b), the x-axis and y-axis represent Time and Loss, respectively, indicating that the method of this invention takes less time than the baseline method when training for the same epoch.

[0017] Figure 3 The reward curves of the method of the present invention over 30 epochs on Flickr30k and MS-COCO are shown, (a) Flickr30k, (b) MS-COCO;

[0018] Figure 4 This is a visualization of the attention weights for a given image-text pair input according to the present invention.

[0019] Figure 5 Visual results of text retrieval for a given image query on the Flickr30K dataset using the method of the present invention;

[0020] Figure 6 This is a visual representation of the image retrieval results for a given text query on the Flickr30K dataset using the method of the present invention. Detailed Implementation

[0021] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.

[0022] This invention provides a cross-modal retrieval method based on relation enhancement strategy gradient and parallel weight control. Figure 1 As shown. The method includes the following steps:

[0023] Step 1: For the image portion of the image-text pair, extract image features using common spatial learning methods. Use BUA (Bottom Up Attention) as the base image network for the image branch, selecting the image from the image-text pair as input to the image branch. For each image I, output a set of salient region features V = {v1, ..., v1}. i ,…,v nLet} represent the image, where: v i Let d represent the feature vector of the i-th salient region being encoded, where i = 1, ..., n, n represents the number of salient regions in the image, and d represents the dimension of the feature vector. The specific steps are as follows:

[0024] Step 1: Use BUA to obtain the features of n salient regions of the input image. For any salient region i, extract its average pooled features to obtain the average pooled feature vector.

[0025] Steps 1 and 2: Apply a fully connected layer to Converting it to a d-dimensional vector is shown in the following formula:

[0026]

[0027] Among them, W v and b v This represents the learnable parameters of the fully connected layer.

[0028] Step 2: For the text portion of the image-text pair, common spatial learning methods are used to extract text features. Word encoding is used to encode each token into a word vector. Then, a bidirectional GRU is used as the base text network for the text branch to transform the word sequence into a text representation U = {u1,…,u...}. i ,…,u m},in: u i Let i represent the feature vector of the i-th text word being encoded, where i = 1, ..., m, and m represents the number of words in the sentence. The specific steps are as follows:

[0029] Step 2.1 For the text portion, sentences containing m words are converted from a fixed vocabulary to one-hot encoding W = {w1, ..., w...} i ,…,w m}, where w i This represents the one-hot encoding of the i-th text word;

[0030] Step 22: Calculate the one-hot encoding w of the i-th text word using a word embedding layer. i Convert to vector Among them: W e It is a learnable parameter matrix of the word embedding layer. This represents the dense encoding of the i-th text word;

[0031] Steps two and three: Input Bi-GRU, as shown in the following formula:

[0032]

[0033] in and These represent the forward and reverse hidden states, respectively.

[0034] Step Two Four: Word Embedding i It is the mean of the two-way hidden states, representing u. i Includes the word w i The central context is shown in the following formula:

[0035]

[0036] Step 3: Model the relationship between arbitrary image region features and all image region features using a self-attention mechanism, and use policy gradient and parallel weight control to guide the weight of all image region features in the final global image features, thus obtaining the global image features v. gl The specific steps are as follows:

[0037] Step 3: Define each image region Where n represents the number of salient regions in the image, the relationship between any image region and all image regions is learned based on the self-attention mechanism as follows:

[0038]

[0039] Among them: W Q b Q W K b K W V and b V f and f represent the learnable parameters of the fully connected layer, respectively. Q f K and f V These represent the corresponding activation functions; Q i K i and V i Let h represent the query vector, key vector, and value vector of the i-th image feature in the self-attention mechanism, respectively; i This represents the relationship between the i-th image feature modeled by the self-attention mechanism and all image features.

[0040] Step 3.2: Obtain global image features using policy gradient and parallel weight control.

[0041] Step 321: Define a discrete action space containing N candidate actions A = {a1, a2, ..., a...} N Define the state s at time t'. t' ={h1,h2,…,h nThe process of generating discrete actions in parallel by an agent parameterized by θ is as follows:

[0042]

[0043] Among them: W μ and b μ f and f represent the learnable parameters of the fully connected layer, respectively. μ This represents the corresponding activation function;

[0044] Step 3.2.2 Introduce Gumbel-softmax to adjust μ based on probability and randomness. t' Sampling is used to select discrete actions:

[0045]

[0046] in: This represents the discrete action selected at time t'. Represents a random policy π θ Select the action at time t' The probability of , where G represents noise that follows a Gumbel distribution;

[0047] Step 3. Obtain the discrete actions at time t' Next, construct a mean of The standard deviation is σ t' The normal distribution is used to sample continuous actions from the normal distribution. The process by which the agent generates a series of actions is as follows:

[0048]

[0049] Among them: W σ and b σ f and f represent the learnable parameters of the fully connected layer, respectively. σ This represents the corresponding activation function. This represents the selected continuous action at time t'. Represents a random policy π θ Select the action at time t' The probability of;

[0050] Steps 3-4: The weights of all image regions are defined as continuous actions. Image global features v gl Defined as:

[0051]

[0052] Where T' represents the number of states in the reinforcement learning process;

[0053] Step 325: The loss functions for discrete and continuous policy gradients are as follows:

[0054]

[0055] Where R(τ) represents the cumulative reward of trajectory τ; B represents the number of samples in the mini-batch.

[0056] Step 4: Model the relationship between arbitrary text word features and all text word features using a self-attention mechanism, and use policy gradient and parallel weight control to guide the weight of all text word features in the final global text feature, thus obtaining the global text feature u. gl The specific steps are as follows:

[0057] Step 41: Define each text word m represents the number of words in the sentence. The relationship between any text word and all text words is learned based on the self-attention mechanism as follows:

[0058]

[0059] Among them: W Q b Q W K b K W V and b V f and f represent the learnable parameters of the fully connected layer, respectively. Q f K and f V These represent the corresponding activation functions;

[0060] Step 4.2: Obtain global text features using policy gradient and parallel weight control.

[0061] Step 421: Define a discrete action space containing N candidate actions A = {a1, a2, ..., a...} N Define the state s at time t'. t' ={h1,h2,…,h m The process of generating discrete actions in parallel by an agent parameterized by θ is as follows:

[0062]

[0063] Among them: W μ and b μ f and f represent the learnable parameters of the fully connected layer, respectively. μ This represents the corresponding activation function;

[0064] Step 422: Introduce Gumbel-softmax to adjust μ based on probability and randomness. t'Sampling is used to select discrete actions:

[0065]

[0066] in: This represents the discrete action selected at time t'. Represents a random policy π θ Select the action at time t' The probability of , where G represents noise that follows a Gumbel distribution;

[0067] Steps 4, 2, and 3: Obtain the discrete actions at time t' Next, construct a mean of The standard deviation is σ t' The normal distribution is used to sample continuous actions from the normal distribution. The process by which the agent generates a series of actions is as follows:

[0068]

[0069] Among them: W σ and b σ f and f represent the learnable parameters of the fully connected layer, respectively. σ This represents the corresponding activation function. This represents the selected continuous action at time t'. Represents a random policy π θ Select the action at time t' The probability of;

[0070] Step 424: The weight of each text word is defined as a continuous action. Text global features u gl Defined as:

[0071]

[0072] Where T' represents the number of states in the reinforcement learning process.

[0073] Step 425: The loss functions for discrete and continuous policy gradients are as follows:

[0074]

[0075] Where R(τ) represents the cumulative reward of trajectory τ; B represents the number of samples in the mini-batch.

[0076] Step 5: Perform similarity matching using global image features and global text features, and supervise the process with hinge loss that includes hard negative sample mining. The optimization objective of discrete continuous policy gradients is used to supervise the weights of image region features and text word features. The specific steps are as follows:

[0077] Step 51: A hinge loss with hard negative sample mining is used as the objective function to optimize image-text matching. When text is used as the query, images matching and non-matching the query are sampled in each mini-batch, forming positive and negative pairs. The similarity of a positive pair should be at least γ higher than that of a negative pair. Similarly, when images are used as the query, positive pairs represent text matching the given query image in each mini-batch, and negative pairs represent text not matching the given query image in each mini-batch. The similarity between negative and positive pairs also satisfies the above constraints. Therefore, the ranking loss... The definition is as follows:

[0078]

[0079] Where: γ represents the residual parameter; <·,·> represent the similarity function between two vectors, usually cosine similarity; This represents the global feature of the j-th text in the mini-batch. Indicates that in mini-batch and Matching global image features This indicates that the mini-batch does not contain... Matching global image features This indicates that the mini-batch does not contain... Global features of the matched text.

[0080] Step 5.2: The cumulative reward R(τ), which is of great guiding significance to the policy gradient optimization process, is defined in this invention as R@K and the average precision (AP) score in the mini-batch. Specifically, each sample is treated as a query to calculate R@1 and AP for all samples in the mini-batch. The cumulative reward R(τ) is defined as follows:

[0081] R(τ)=R@1+AP (33)

[0082] Step 5.3: The policy gradient loss functions for the image and text branches are defined as follows:

[0083]

[0084] Step 54: Optimize the method of the present invention by minimizing the following objective function:

[0085]

[0086] Experimental results:

[0087] For image features, this invention references BUA and SCAN to set up the image feature extraction network. Each image is represented by 36 salient regions, and a 2048-dimensional feature is extracted from each salient region. For text features, this invention sets the word embedding dimension to 300 and the dimensions d of both the image latent space and the text latent space to 1024. In all experiments, this invention sets the hyperparameter γ of formula (30) to 0.2 and the number of candidate actions in the discrete action space to N = 500. This invention trains PGRR for 30 epochs on the training sets of the Flickr30K dataset and the MS-COCO dataset, respectively, using the Adam optimizer. PGRR is validated on the corresponding validation set for each epoch. The mini-batch size of this invention is 128, the initial learning rate is set to 0.0002, and the learning rate decays by 10% every 15 epochs. For evaluation on the test set, this invention uses the recall at r (R@r, r = 1, 5, 10) and rSum to measure the performance of PGRR. R@r represents whether, given a query image or text, the first r retrieved samples contain at least one correct text or image. rSum represents the sum of all R@r values ​​for the image retrieval text task and the text retrieval image task.

[0088]

[0089] Figure 2 The training process of this invention over 30 epochs on Flickr30k is illustrated. DCPG represents the baseline method of this invention (i.e., policy gradient control), and PGRR represents the method proposed in this invention (i.e., policy gradient and parallel weight control). In Figure (a), the x and y axes represent Step and Loss, respectively, showing that the method of this invention converges to a lower Loss under the same epoch conditions. In Figure (b), the x and y axes represent Time and Loss, respectively, showing that the method of this invention takes less time than the baseline method to train for the same epoch.

[0090] Figure 3 The reward curves of this invention over 30 epochs are shown on Flickr30k (Fig. (a)) and MS-COCO (Fig. (b)). Under different conditions, the reward of this invention can gradually increase and eventually converge during the training process, demonstrating its stability in terms of the reward curve.

[0091] Figure 4This paper presents a visualization of the attention weights applied to a given image-text pair of inputs according to the present invention. The invention controls the weights of all salient region features to generate the final image global vector and controls the weights of all word features to generate the final text global vector. From left to right, the first column represents the input image. The second column represents the attention map of the image. The third column represents the weights of word features in the final text global feature vector. It can be seen that by assigning higher weights to important regions in the image and higher weights to important words in the text, the present invention can generate more robust global features containing accurate semantic information for cross-modal retrieval.

[0092] Figure 5 This paper presents a visual representation of the text retrieval results for a given image query on the Flickr30K dataset, showcasing the achievements of this invention. Figure 6 This document presents a visualization of the image retrieval results of this invention on the Flickr30K dataset, given a text query. Samples that actually match are marked in green.

[0093] Table 1 presents the quantitative evaluation results of this invention and mainstream cross-modal retrieval methods on the MS-COCO dataset using the R@1, R@5, and R@10 metrics. For a more comprehensive comparison, this invention was tested on MS-COCO using two test set settings (1K and 5K). It can be seen that this invention achieves state-of-the-art performance on most metrics in image retrieval and sentence retrieval tasks. In particular, under the 5K setting, this invention demonstrates significant improvement compared to previous methods.

[0094] Table 1 compares the experimental results with the current mainstream cross-modal retrieval results on the MS-COCO database.

[0095]

[0096] Table 2 presents the comparative results of this invention on the Flickr30K dataset. It can be seen that this invention outperforms other methods in most evaluation metrics, demonstrating its superiority.

[0097] Table 2 compares the experimental results with the best current cross-modal retrieval results on the Flickr30K database.

[0098]

Claims

1. A cross-modal retrieval method based on relation enhancement strategy gradient and parallel weight control, characterized in that... The method includes the following steps: Step 1: For the image portion of the image-text pair, use BUA as the base image network for the image branch. Select the image in the image-text pair as the input of the image branch, and output a set of features V = {v1, ..., v1} for each image I. i , ..., v n Let} represent the image, where: v i d represents the feature vector of the i-th salient region being encoded, where i = 1, ..., n, n represents the number of salient regions in the image, and d represents the dimension of the feature vector; Step 2: For the text portion of the image-text pair, use word encoding to encode each token into a word vector. Then, use a bidirectional GRU as the base text network for the text branches to transform the word sequence into a text representation U = {u1, ..., u1}. i ,…,u m },in: u i The feature vector represents the i-th word of the encoded text, where i = 1, ..., m, and m represents the number of words in the sentence; Step 3: Model the relationship between arbitrary image region features and all image region features using a self-attention mechanism, and use policy gradient and parallel weight control to guide the weight of all image region features in the final global image features, thus obtaining the global image features v. gl ; Step 4: Model the relationship between arbitrary text word features and all text word features using a self-attention mechanism, and use policy gradient and parallel weight control to guide the weight of all text word features in the final global text feature, thus obtaining the global text feature u. gl ; Step 5: Perform similarity matching using global image features and global text features, and supervise the process with hinge loss with hard negative sample mining. Use the optimization objective of discrete continuous policy gradient to supervise the weights of image region features and text word features.

2. The cross-modal retrieval method based on relation enhancement strategy gradient and parallel weight control according to claim 1, characterized in that... The specific steps of step one are as follows: Step 1: Use BUA to obtain the features of n salient regions of the input image. For any salient region i, extract its average pooled features to obtain the average pooled feature vector. Steps 1 and 2: Apply a fully connected layer to Converting it to a d-dimensional vector is shown in the following formula: Among them, W v and b v This represents the learnable parameters of the fully connected layer.

3. The cross-modal retrieval method based on relation enhancement strategy gradient and parallel weight control according to claim 1, characterized in that... The specific steps of step two are as follows: Step 2.1 For the text portion, sentences containing m words are converted from a fixed vocabulary to one-hot encoding W = {w1, ..., w...} i ,…,w m }, where w i This represents the one-hot encoding of the i-th text word; Step 22: Calculate the one-hot encoding w of the i-th text word using a word embedding layer. i Convert to vector Among them: W e It is a learnable parameter matrix of the word embedding layer. This represents the dense encoding of the i-th text word; Steps two and three: Input Bi-GRU, as shown in the following formula: in and These represent the forward and reverse hidden states, respectively. Step Two Four: Word Embedding i It is the mean of the two-way hidden states, representing u. i Includes the word w i The central context is shown in the following formula:

4. The cross-modal retrieval method based on relation enhancement strategy gradient and parallel weight control according to claim 1, characterized in that... The specific steps of step three are as follows: Step 3: Define each image region i = 1, ..., n, where n represents the number of salient regions in the image. The relationship between any image region and all image regions is learned based on the self-attention mechanism as follows: Among them: W Q b Q W K b K W V and b V f and f represent the learnable parameters of the fully connected layer, respectively. Q f K and f V These represent the corresponding activation functions; Q i K i and V i Let h represent the query vector, key vector, and value vector of the i-th image feature in the self-attention mechanism, respectively; i This represents the relationship between the i-th image feature modeled by the self-attention mechanism and all image features; Step 3.2: Obtain global image features using policy gradient and parallel weight control. Step 321: Define the discrete action space containing N candidate actions A = {a1, a2, ... a3}. N Define the state s at time t′. t′ ={h1, h2, ..., h n The process of generating discrete actions in parallel by an agent parameterized by θ is as follows: Among them: W μ and b μ f and f represent the learnable parameters of the fully connected layer, respectively. μ This represents the corresponding activation function; Step 3.2.2 Introduce Gumbel-softmax to adjust μ based on probability and randomness. t′ Sampling is used to select discrete actions: in: This represents the discrete action selected at time t′. Represents a random policy π θ Select the action at time t′ The probability of , where G represents noise that follows a Gumbel distribution; Step 3. Obtain the discrete actions at time t′. Next, construct a mean of The standard deviation is σ t′ The normal distribution is used to sample continuous actions from the normal distribution. The process by which the agent generates a series of actions is as follows: Among them: W σ and b σ f and f represent the learnable parameters of the fully connected layer, respectively. σ This represents the corresponding activation function. This represents the selected continuous action at time t′. Represents a random policy π θ Select the action at time t′ The probability of; Steps 3-4: The weights of all image regions are defined as continuous actions. Image global features v gl Defined as: Where T′ represents the number of states in the reinforcement learning process; Step 325: The loss functions for discrete and continuous policy gradients are as follows: Where R(τ) represents the cumulative reward of trajectory τ; B represents the number of samples in the mini-batch.

5. The cross-modal retrieval method based on relation enhancement strategy gradient and parallel weight control according to claim 1, characterized in that... The specific steps of step four are as follows: Step 41: Define each text word i = 1, ..., m, where m represents the number of words in the sentence. The relationship between any text word and all text words is learned based on the self-attention mechanism as follows: Among them: W Q b Q W K b K W V and b V f and f represent the learnable parameters of the fully connected layer, respectively. Q f K and f V These represent the corresponding activation functions; Step 4.2: Obtain global text features using policy gradient and parallel weight control. Step 421: Define a discrete action space containing N candidate actions A = {a1, a2, ..., a...} N Define the state s at time t′. t′ ={h1, h2, ..., h m The process of generating discrete actions in parallel by an agent parameterized by θ is as follows: Among them: W μ and b μ f and f represent the learnable parameters of the fully connected layer, respectively. μ This represents the corresponding activation function; Step 422: Introduce Gumbel-softmax to adjust μ based on probability and randomness. t′ Sampling is used to select discrete actions: in: This represents the discrete action selected at time t′. Represents a random policy π θ Select the action at time t′ The probability of , where G represents noise that follows a Gumbel distribution; Steps 4, 2, and 3: Obtain the discrete actions at time t′. Next, construct a mean of The standard deviation is σ t′ The normal distribution is used to sample continuous actions from the normal distribution. The process by which the agent generates a series of actions is as follows: Where Wσ and bσ represent the learnable parameters of the fully connected layer, and fσ represents the corresponding activation function. This represents the selected continuous action at time t′. Represents a random policy π θ Select the action at time t′ The probability of; Step 424: The weight of each text word is defined as a continuous action. Text global features u gl Defined as: Where T′ represents the number of states in the reinforcement learning process; Step 425: The loss functions for discrete and continuous policy gradients are as follows: Where R(τ) represents the cumulative reward of trajectory τ; B represents the number of samples in the mini-batch.

6. The cross-modal retrieval method based on relation enhancement strategy gradient and parallel weight control according to claim 4, characterized in that... The specific steps of step five are as follows: Step 51: Use hinge loss with hard negative sample mining as the objective function to optimize image-text matching. When text is used as the query, in each mini-batch, images that match and do not match the query are sampled separately to form positive and negative pairs. When images are used as the query, positive pairs represent text that matches the given query image in each mini-batch, and negative pairs represent text that does not match the given query image in each mini-batch. Ranking loss... The definition is as follows: Where: γ represents the residual parameter; <·, ·> represent the similarity function between the two vectors; This represents the j-th global feature of the text in the mini-batch. Indicates that in mini-batch and Matching global image features This indicates that the mini-batch does not contain... Matching global image features This indicates that the mini-batch does not contain... Global features of the matched text; Step 5.2: Treat each sample as a query to calculate R@1 and AP for all samples in the mini-batch. The cumulative reward R(τ) is defined as follows: R(τ) = R@1 + AP; Step 5.3: The policy gradient loss functions for the image and text branches are defined as follows: Step 54: Optimize the method of the present invention by minimizing the following objective function:

Images