Combined image retrieval method based on multi-granularity object-level matching fusion

Abstract

The invention discloses a combined image retrieval method based on multi-granularity object-level matching fusion. The method comprises the following steps: constructing a visual refinement device sub-network formed by connecting an image feature extraction module and a clustering merging module in series; constructing a cross-modal interaction sub-network formed by connecting an image feature extraction module and a cross-modal interaction module in series; constructing a combined image retrieval network which is formed by connecting a multi-granularity image processing sub-network and a combined query input fusion sub-network in series and then connecting the multi-granularity image processing sub-network and the combined query input fusion sub-network in series with a cross-modal interaction sub-network in series; using the combined image retrieval training set to train a combined image retrieval network; and retrieving a target image corresponding to the combined query input. According to the method, the retrieval performance of combined image retrieval is improved, and the method can be used for electronic commerce, wisdom education and security and protection monitoring.

Description

Technical Field

[0001] This invention belongs to the field of multimodal image retrieval and further relates to a combined image retrieval method based on multi-granular object-level matching and fusion. This invention can be used in e-commerce, smart education, and security monitoring to retrieve target images from an image library using reference images and text information. Background Art

[0002] With the rapid development of internet technology and the widespread adoption of digital devices, image data has exploded in volume. Image retrieval technology has been widely applied in fields such as search engines, e-commerce, and smart education. However, traditional image retrieval methods face numerous challenges, one of which is the difficulty in accurately capturing user search intent solely through textual descriptions or similar images. This is primarily due to differences in people's expressive abilities and habits when using textual descriptions, which can lead to information omissions and semantic confusion in the transmission of information. Furthermore, with the explosive growth of image data, the number of similar images has increased dramatically, further exacerbating the discrepancy between retrieval results and users' actual needs, resulting in retrieval results filled with many similar but undesirable images. To overcome these challenges, combined image retrieval techniques aim to allow users to first use existing images as an initial reference and then combine them with text to further refine the desired modifications to the reference image. Combined image retrieval methods typically involve three steps: feature extraction, cross-modal alignment and interaction, and target image retrieval. The core issue lies in cross-modal semantic alignment and interaction. Although existing combined image retrieval methods have achieved promising results, they remain limited in understanding the multiple objects and complex changes in text modifications to the reference image. On the one hand, due to the lack of detailed object-level visual features, the model struggles to effectively learn to detect fine-grained visual objects in images referenced by text, and lacks clarity about the image objects and attributes to be modified. On the other hand, the model fails to adequately model cross-modal attention for the combined input and target images, making it difficult to capture changes in visual objects and attributes from the reference image to the target image. Combination image retrieval requires two key issues: "Which objects in the image are modified by the text?" and "How do the text guide changes in image objects?" Therefore, effective cross-modal semantic alignment and interaction are necessary to further address performance bottlenecks.

[0003] The University of Electronic Science and Technology of China (USTC) has disclosed a combined image retrieval method in its patent application, "A Hierarchical Fusion Combined Query Image Retrieval Method" (Application Number: CN 202111348833.3, Application Publication Number: CN 114048340 A). This method first extracts image and text features separately based on a hierarchical feature extraction module based on a scene graph. It then uses a multimodal feature hierarchical fusion module to fuse global, local, and scene graph features using a gating and attention mechanism. A triplet metric function is used as the loss function to train a hierarchical fusion learning model based on the scene graph. Finally, a cluster matching model is constructed to match the fused features with the target image encoding in the target image encoding database to complete combined image retrieval. This method has two shortcomings: First, although it considers the fusion of text descriptions and visual features at different levels, the multiple levels of visual features are redundant and interfere with each other, making it difficult to accurately match semantic information in the text with image features. Second, although this method processes image and text information by constructing different modules, it fails to fully model cross-modal attention for the combined input and target images, and cannot accurately model the changes in visual objects and attributes from the reference image to the target image.

[0004] In their paper "Bi-directional Training for Composed Image Retrieval via Text Prompt Learning" (IEEE / CVF Winter Conference on Applications of Computer Vision, 2024), Liu et al. presented a combined image retrieval method based on cross-modal contrastive learning. Based on the CLIP4Cir method and a BLIP pre-trained model, this method proposes a bidirectional training scheme for combined image retrieval, leveraging reverse queries to improve model performance. Learnable markers are added before modifying the text to specify the query direction, generating either forward or reverse text embeddings for a given text. After obtaining the reverse text embeddings, the target image and reverse text embeddings are combined to form a reverse query pair by swapping the image order. The extracted image features are element-wise summed with the text features, and a combiner network is trained to achieve combined image retrieval. However, this method uses a single global feature embedding for both image and text, which lacks clarity about the image object and attributes being modified, making it difficult for the model to capture fine-grained semantic associations between the image and text. Furthermore, since it only uses simple element-wise addition to fine-tune the visual language pre-trained model, the model's performance still has significant room for improvement. Summary of the Invention

[0005] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose a combined image retrieval method based on multi-granularity object-level matching fusion. This method is used to address the lack of refined object-level visual features in the prior art, making it difficult for the model to effectively learn to detect fine-grained visual objects in images related to text, unclear about the image objects and attributes to be modified, and insufficiently modeling cross-modal attention for the combined input and target images. This makes it difficult to capture the changes in visual objects and attributes from the reference image to the target image, as well as complex semantic relationships, thus limiting model performance.

[0006] The idea for achieving the purpose of the present invention is to solve the problem of the model understanding "which objects in the image the text is to modify". The present invention extracts the reference image bounding box related to the text in the reference image, and designs a visual refiner subnetwork to refine the reference image bounding box information to obtain object-level visual features. The multi-granularity image processing subnetwork processes the diverse semantic information in the reference image at multiple granularities, thereby enhancing the model's understanding of the image attribute details that the text is to modify. This solves the problem that the existing technology has difficulty in capturing fine-grained semantic association information between images and text. For the model to understand the problem of "how the text guides the image objects to change", the key lies in in-depth modeling of the combined input and target image. The present invention designs a cross-modal interaction subnetwork based on hybrid attention to deeply model the combined input and target image. This solves the problem that the existing technology fails to fully model cross-modal attention for the combined input and target image, resulting in the inability to accurately map the semantic information from the combined input to the corresponding area in the target image, thereby improving the accuracy of the combined image retrieval model.

[0007] According to the above ideas, the technical solution of the present invention includes the following steps:

[0008] Step 1: Construct a visual refiner subnetwork consisting of an image feature extraction module and a clustering merging module in series;

[0009] Step 2: Construct a cross-modal interaction sub-network consisting of an image feature extraction module and a cross-modal interaction module in series;

[0010] Step 3: Construct a combined image retrieval network consisting of a multi-granularity image processing subnetwork connected in series with a combined query input fusion subnetwork, and then connected in series with a cross-modal interaction subnetwork;

[0011] Step 4, using the combined image retrieval training set to train the combined image retrieval network;

[0012] Step 5: Retrieve the target image corresponding to the combined query input:

[0013] The combined query data consisting of the reference image to be queried and the modified text is input into the trained combined image retrieval network, and the retrieval of the target image corresponding to the semantics of the input data to be queried is output.

[0014] Furthermore, the image feature extraction module is implemented using a Vision Transformer network with 12 layers. The clustering and merging module consists of a clustering module and a merging module, wherein the clustering module uses the K-Means algorithm for clustering operations, and the merging module performs dynamic merging based on a cross-attention mechanism.

[0015] Furthermore, the cross-modal interaction sub-network is implemented using the Vision Transformer and BERT network structures, and the combined query-target contrast loss and the combined query-target matching loss are calculated in the cross-modal interaction module; the cross-modal interaction module pre-reduces the modal heterogeneity of the combined query input and the target image by calculating the multi-head attention of each layer; the multi-head attention of each layer t The calculation formula is as follows:

[0016] head t =(1-λ(F t ))softmax(Q t K t T )V t +λ(F t )softmax(Q t K c T )V c

[0017] Among them, F t represents the target image embedding, λ(F t ) represents the sum of the normalized attention weights on the target image embedding key-value pair vector, softmax(·) represents the activation function that normalizes the vector to a probability distribution vector, Q t , K t and V t They represent the query, key, and value parameters after the target image is embedded by the image feature extraction module in the cross-modal interaction sub-network, and the superscript T represents the transposition operation, K c and V c They represent the key and value parameters mapped by the coefficient matrix after the combined query is embedded by the fusion module.

[0018] Furthermore, the multi-granularity image processing subnetwork is composed of a visual refiner subnetwork and an image feature extraction module in parallel, which are then connected in series with a multi-granularity image fusion module; the multi-granularity image fusion module uses a mapping layer to perform fusion operations on images of different granularities.

[0019] Furthermore, the combined query input fusion subnetwork is composed of a text feature extraction module and a combined query input fusion module connected in series, wherein both the text feature extraction module and the combined query input fusion module adopt a 6-layer BERT network structure; the combined query input fusion module adopts self-attention calculation and cross-attention calculation;

[0020] The self-attention calculation Attn(·) is implemented by the following formula:

[0021]

[0022] Among them, Q x , K x and V x represents the query, key, and value parameters of the modified text after being mapped by the coefficient matrix embedded in the combined query input fusion module. d represents the dimension of the modified text sequence vector.

[0023] The cross attention calculation CAttn(·) is implemented by the following formula:

[0024]

[0025] Among them, K y and V y They represent the key and value parameters of the multi-granularity visual embedding sequence vector obtained after the multi-granularity image processing sub-network.

[0026] Furthermore, the combined image retrieval training set refers to selecting at least 30,000 groups of samples, each group of samples containing a one-to-one correspondence between a reference image, modified text, a reference image bounding box, and a target image, wherein the content of the modified text is a natural language description of the reference image attributes and changes from the reference image to the target image, and the reference image bounding box is a bounding box for extracting the modified text and its corresponding reference image area.

[0027] Furthermore, the training of the combined image retrieval network refers to inputting the combined image retrieval training set into the combined image retrieval network, and using the gradient descent method to iteratively update the weight values ​​of the network parameters until the combined query-target contrast loss function and the combined query-target matching loss function of the cross-modal interaction module in the cross-modal interaction subnetwork converge at the same time, thereby obtaining a trained combined image retrieval network.

[0028] The combined query-target contrast loss function It is obtained by the following formula:

[0029]

[0030] Where B represents the amount of data in a single batch of the training set input to the combined image retrieval network; i and j represent the i-th and j-th sample vectors in the combined image retrieval training set in a single batch, respectively; exp{·} represents the exponential operation with the natural constant e as the base; cos(·) represents the cosine operation; represents the mth combined query feature mapped after the combined query embedding obtained by the combined query input fusion module; It represents the nth target image feature mapped after the target image is embedded through the image feature extraction module in the cross-modal interaction sub-network.

[0031] The combined query-target matching loss function It is obtained by the following formula:

[0032]

[0033] in, represents the combined query embedding Q obtained by the combined query input fusion module and the target image embedding T obtained by the image feature extraction module in the cross-modal interaction sub-network; H(·) represents the cross entropy loss function, which is used to measure the true combined query-target matching label y qtm Combined query-target matching label p with prediction qtm The difference between qtm represents the probability of combined query-target matching predicted by softmax in the combined query input fusion module; qtm Represents the true combined query-target matching label, using 0 or 1 binary values ​​to indicate match or mismatch.

[0034] Compared with the prior art, the present invention has the following advantages:

[0035] First, the present invention constructs a visual refiner subnetwork in the combined image retrieval network, which can refine the reference image bounding box information related to the text, so that the model can process semantically perceived visual objects and integrate the visual object information in multiple reference image bounding boxes, overcoming the problem that the existing technology has difficulty in capturing fine-grained semantic association information between images and text, so that the model of the present invention can clearly understand "which objects on the image the text is going to modify", enhance the model's understanding of the details of the image attributes that the text is going to modify, and promote cross-modal semantic matching.

[0036] Second, the present invention constructs a cross-modal interaction subnetwork in the combined image retrieval network, calculates multi-head attention in each layer through mixed attention to pre-reduce modal heterogeneity, and performs deep modeling of the combined input and target image, overcoming the defects of the existing technology that fails to fully model cross-modal attention for the combined input and target image, and cannot accurately map the semantic information from the combined input to the corresponding area in the target image. The model of the present invention is able to learn "how text guides visual objects to change", promote cross-modal semantic interaction, and thus improve the accuracy of the combined image retrieval method. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flow chart of an embodiment of the present invention;

[0038] Figure 2 is a schematic diagram of the clustering and merging module in the visual refiner subnetwork of the present invention;

[0039] Figure 3 is a schematic diagram of a cross-modal interaction module in a cross-modal interaction sub-network of the present invention;

[0040] Figure 4 It is a schematic diagram of the combined image retrieval network structure of the present invention. DETAILED DESCRIPTION

[0041] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0042] Reference Figure 1 , the specific implementation steps of the embodiment of the present invention are further described in detail.

[0043] Step 1: Construct a visual refiner subnetwork consisting of an image feature extraction module and a clustering merging module in series.

[0044] Reference Figure 2 , the designed visual refiner subnetwork and the clustering merging module therein are further described in detail.

[0045] The image feature extraction module in this embodiment of the present invention is implemented using the Vision Transformer network of the prior art, with 12 layers and an output vector dimension of 1408. The clustering and merging module consists of a clustering module and a merging module. The clustering module uses the K-Means algorithm for clustering, and the merging module dynamically merges image features of the same category based on a cross-attention mechanism.

[0046] Step 2: Construct a cross-modal interaction sub-network consisting of an image feature extraction module and a cross-modal interaction module in series.

[0047] Reference Figure 3, and further describe the designed cross-modal interaction sub-network and the cross-modal interaction module therein in detail.

[0048] The cross-modal interaction subnetwork of the embodiment of the present invention is implemented using the Vision Transformer and BERT network structures of the prior art, and the combined query-target contrast loss and combined query-target matching loss are calculated in the cross-modal interaction module. The cross-modal interaction module uses hybrid attention to calculate multi-head attention in each layer to pre-reduce the modal heterogeneity of the combined query input and target image. Since the Vision Transformer and BERT used are both based on the Transformer architecture, the designed hybrid attention is also based on the Transformer attention mechanism.

[0049] The target image is feature extracted through the image feature extraction module in the cross-modal interaction sub-network to obtain the target image embedded sequence vector F t , and through the coefficient matrix W t q , W t k , W t v F t Mapping to query Q t , key and value parameters K t and V t The combined query is embedded in the combined query input fusion sub-network through the combined query input fusion module to perform feature fusion. q After that, through the coefficient matrix Mapping to query Q c , key and value parameters K c and V c ; Each layer of multi-head attention head t The calculation formula is as follows:

[0050] head t =(1-λ(F t ))softmax(Q t K t T )V t +λ(F t )softmax(Q t K c T )V c

[0051] Among them, λ(F t) represents the sum of the normalized attention weights on the target image embedding key-value pair vector, softmax(·) represents the activation function that normalizes the vector into a probability distribution vector, and the superscript T represents the transpose operation.

[0052] Step 3: Build a combined image retrieval network.

[0053] Reference Figure 4 , a further detailed description of the constructed combined image retrieval network is given.

[0054] The combined image retrieval network consists of a multi-granularity image processing subnetwork connected in series with a combined query input fusion subnetwork, and then connected in series with a cross-modal interaction subnetwork.

[0055] The combined query input fusion subnetwork is composed of a text feature extraction module and a combined query input fusion module connected in series, wherein both the text feature extraction module and the combined query input fusion module adopt the BERT network structure of the existing technology, with 6 network layers and an output vector dimension of 768.

[0056] The combined query input fusion module adopts self-attention calculation and cross-attention calculation; the self-attention calculation Attn(·) is implemented by the following formula:

[0057]

[0058] Among them, Q x , K x and V x Represents the coefficient matrix of the modified text embedded by the combined query input fusion module The query, key, and value parameters after mapping, d represents the dimension of the modified text sequence vector.

[0059] The cross attention calculation CAttn(·) is implemented as follows:

[0060]

[0061] Among them, K y and V y They represent the key and value parameters of the multi-granularity visual embedding sequence vector obtained after the multi-granularity image processing sub-network.

[0062] Step 4: Generate a combined image retrieval training set.

[0063] In an embodiment of the present invention, 30,000 groups of samples were selected from a public combined image retrieval public data set. Each group of samples contained a one-to-one correspondence between a reference image, modified text, a reference image bounding box, and a target image. The content of the modified text was a natural language description of the reference image attributes and changes from the reference image to the target image, and the reference image bounding box was a bounding box for extracting the modified text and its corresponding reference image area.

[0064] The reference image bounding box is extracted by the large language model and the visual positioning model, which can be obtained by the following steps.

[0065] First, the large language model GPT4 is used to extract key words related to the reference image attributes and describing the changes from the modified text, and obtain keywords of the fine-grained image attributes of each modified text, such as but not limited to "short sleeves", "long skirt", "dog", "red", "tight", "with buttons", etc.

[0066] The reference image and the extracted modified text keywords are then fed into the GroundingDINO visual localization model to extract the bounding boxes of the corresponding reference image regions. Since modified text typically contains both the image attributes of the reference image and the specified changes relative to the target image, not every keyword is necessarily related to the reference image. Therefore, for each extracted keyword, all image bounding boxes exceeding the filtering threshold are selected to extract the image bounding boxes of the potentially modified regions of each image in the dataset.

[0067] Step 5: Train the combined image retrieval network.

[0068] The combined image retrieval training set is input into the combined image retrieval network. The image feature extraction module and the visual refiner module in the visual refiner subnetwork are forward propagated to extract the global reference image embedding and the fine-grained object-level reference image embedding, respectively. The multi-granular image fusion module is forward propagated to obtain the multi-granular visual embedding, and the text feature extraction module is forward propagated to extract the text embedding. The text-image combined input fusion module is forward propagated to obtain the unified combined query embedding, and the target image feature extraction module is forward propagated to extract the target image embedding. A combined query-target contrastive loss is calculated to minimize the distance between the combined query features and the target image features. The cross-modal interaction module is forward propagated to use the tokens embedded in the multimodal encoder output as the joint representation of the combined query-target image pair. After passing through the fully connected layer (FC), the matching probability is calculated. The combined query-target matching loss is calculated, and the network parameter weights are iteratively updated using gradient descent until the combined query-target contrastive loss function and the combined query-target matching loss function of the cross-modal interaction module in the cross-modal interaction subnetwork converge simultaneously, resulting in the trained combined image retrieval network.

[0069] The combined query-target matching loss This is achieved by:

[0070]

[0071] Among them, y qtm A two-dimensional one-hot vector representing the true class label.

[0072] The complete pre-training objectives of the combined image retrieval network are as follows:

[0073]

[0074] Step 6: Retrieve the target image corresponding to the combined query input.

[0075] The combined query data consisting of the reference image to be queried and the modified text is sent to the trained combined image retrieval network for feature extraction and interactive fusion to form a combined query feature. At the same time, all image features in the dataset are pre-extracted. By calculating the matching scores between the combined query feature and all image features in the dataset and sorting them, the candidate target image with the highest matching score is selected as the final combined image retrieval result, completing the retrieval of the target image corresponding to the semantics of the combined input to be queried.

[0076] The above description is only a specific example of the present invention and does not constitute any limitation to the present invention. Obviously, for professionals in this field, after understanding the content and principles of the present invention, it is possible to make various modifications and changes in form and details without departing from the principles and structure of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

[0077] It should be noted that the step numbers in the specification and claims of the present invention are only for the purpose of clearly describing the embodiments of the present invention and facilitating understanding, and the order of the step numbers is not limited.

Claims

1. A combined image retrieval method based on multi-granularity object-level matching fusion, characterized in that: The visual refiner subnetwork and the cross-modal interaction subnetwork are constructed separately; the steps of the combined image retrieval method include the following: Step 1: Construct a visual refiner subnetwork consisting of an image feature extraction module and a clustering merging module in series; Step 2: Construct a cross-modal interaction sub-network consisting of an image feature extraction module and a cross-modal interaction module in series; Step 3: Construct a combined image retrieval network consisting of a multi-granularity image processing subnetwork connected in series with a combined query input fusion subnetwork, and then connected in series with a cross-modal interaction subnetwork; Step 4, using the combined image retrieval training set to train the combined image retrieval network; Step 5: Retrieve the target image corresponding to the combined query input: The combined query data consisting of the reference image to be queried and the modified text is input into the trained combined image retrieval network, and the retrieval of the target image corresponding to the semantics of the input data to be queried is output.

2. The combined image retrieval method according to claim 1, characterized in that: The image feature extraction module described in step 1 is implemented using the Vision Transformer network with 12 network layers.

3. The combined image retrieval method according to claim 1, wherein: The clustering merging module described in step 1 consists of a clustering module and a merging module, wherein the clustering module uses the K-Means algorithm to perform clustering operations, and the merging module performs dynamic merging based on the cross-attention mechanism.

4. The combined image retrieval method according to claim 1, characterized in that: The cross-modal interaction sub-network in step 2 is implemented using the Vision Transformer and BERT network structures, and the combined query-target contrast loss and combined query-target matching loss are calculated in the cross-modal interaction module; the cross-modal interaction module reduces the modal heterogeneity of the combined query input and the target image in advance by calculating the multi-head attention of each layer; the multi-head attention head of each layer t The calculation formula is as follows: head t =(1-λ(F t ))softmax(Q t K t T )V t +λ(F t )softmax(Q t K c T )V c Among them, F t represents the target image embedding, λ(F t ) represents the sum of the normalized attention weights on the target image embedding key-value pair vector, softmax(·) represents the activation function that normalizes the vector to a probability distribution vector, Q t , K t and V t They represent the query, key, and value parameters after the target image is embedded by the image feature extraction module in the cross-modal interaction sub-network, and the superscript T represents the transposition operation, K c and V c They represent the key and value parameters mapped by the coefficient matrix after the combined query is embedded by the fusion module.

5. The combined image retrieval method according to claim 4, characterized in that: The multi-granularity image processing subnetwork in step 3 is composed of a visual refiner subnetwork and an image feature extraction module in parallel, and then connected in series with a multi-granularity image fusion module; the multi-granularity image fusion module uses a mapping layer to perform a fusion operation on images of different granularities.

6. The combined image retrieval method according to claim 5, characterized in that: The combined query input fusion subnetwork described in step 3 is composed of a text feature extraction module and a combined query input fusion module connected in series, wherein both the text feature extraction module and the combined query input fusion module adopt a 6-layer BERT network structure; the combined query input fusion module adopts self-attention calculation and cross-attention calculation; The self-attention calculation Attn(·) is implemented by the following formula: Among them, Q x , K x and V x represents the query, key, and value parameters of the modified text after being mapped by the coefficient matrix embedded in the combined query input fusion module. d represents the dimension of the modified text sequence vector. The cross attention calculation CAttn(·) is implemented by the following formula: Among them, K y and V y They represent the key and value parameters of the multi-granularity visual embedding sequence vector obtained after the multi-granularity image processing sub-network.

7. The combined image retrieval method according to claim 1, characterized in that: The combined image retrieval training set described in step 4 refers to selecting at least 30,000 groups of samples, each group of samples containing a one-to-one correspondence between a reference image, modified text, a reference image bounding box, and a target image, wherein the content of the modified text is a natural language description of the reference image attributes and changes from the reference image to the target image, and the reference image bounding box is a bounding box for extracting the modified text and its corresponding reference image area.

8. The combined image retrieval method according to claim 1, wherein: The training of the combined image retrieval network in step 4 refers to inputting the combined image retrieval training set into the combined image retrieval network, and iteratively updating the weight values ​​of the network parameters using the gradient descent method until the combined query-target contrast loss function and the combined query-target matching loss function of the cross-modal interaction module in the cross-modal interaction subnetwork converge at the same time, thereby obtaining a trained combined image retrieval network.

9. The combined image retrieval method according to claim 8, characterized in that: The combined query-target contrast loss function It is obtained by the following formula: Where B represents the amount of data in a single batch of the training set input to the combined image retrieval network; i and j represent the i-th and j-th sample vectors in the combined image retrieval training set in a single batch, respectively; exp{·} represents the exponential operation with the natural constant e as the base; cos(·) represents the cosine operation; represents the mth combined query feature mapped after the combined query embedding obtained by the combined query input fusion module; It represents the nth target image feature mapped after the target image is embedded through the image feature extraction module in the cross-modal interaction sub-network.

10. The combined image retrieval method according to claim 9, characterized in that: The combined query-target matching loss function It is obtained by the following formula: in, represents the combined query embedding Q obtained by the combined query input fusion module and the target image embedding T obtained by the image feature extraction module in the cross-modal interaction sub-network; H(·) represents the cross entropy loss function, which is used to measure the true combined query-target matching label y qtm Combined query-target matching label p with prediction qtm The difference between qtm represents the probability of combined query-target matching predicted by softmax in the combined query input fusion module; qtm Represents the true combined query-target matching label, using 0 or 1 binary values ​​to indicate match or mismatch.

Images