A target-guided multimodal combined image retrieval method and system

Abstract

This invention belongs to the field of image information retrieval technology and provides a target-guided multimodal combined image retrieval method-level system. The system comprises an attribute feature extraction module, a multimodal fusion module guided by the target-query relationship, and a metric learning module guided by the target similarity distribution. The attribute feature extraction module extracts unified attribute features for the reference image, modified text, and target image. The latter two modules, using target image information as a guide, perform multimodal fusion and metric learning, respectively, achieving excellent multimodal combined image retrieval performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of image information retrieval, and in particular relates to a target-guided multi-modal combined image retrieval method and system. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Compared to traditional image retrieval tasks based on single-modal queries, multimodal combined image retrieval is a more flexible retrieval paradigm, allowing users to retrieve images using multimodal query conditions. The reference image represents the user's overall retrieval requirement, while the modified text reflects the user's need to modify locally unsatisfactory attributes. The key to this task lies in effective multimodal fusion and accurate metric learning.

[0004] For multimodal fusion, existing work has designed a variety of neural networks to fuse reference images and modified text, but all of them have neglected to model the conflicting relationship between the two. In multimodal queries, the essence lies in modeling the conflicting relationship between the reference image and the modified text, thereby retaining the visual information that needs to be retained in the reference image and replacing the parts of the modified text that need to be modified. As for the metric learning part, most existing work uses a classification loss function based on the training batch for optimization. However, this method only treats the target image as a positive example, while all other images in the training batch are treated as negative examples. This results in the existing model being unable to reasonably model the metric relationship between the multimodal query and other images in the training batch, which will affect the performance of multimodal combined image retrieval. Summary of the Invention

[0005] In order to solve at least one technical problem existing in the above-mentioned background technology, the present invention provides a target-guided multimodal combined image retrieval method and system, which, by introducing target image information, can perform conflict relationship modeling in the multimodal fusion stage and perform adaptive matching modeling in metric learning, thereby greatly improving the performance of multimodal combined image retrieval.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A first aspect of the present invention provides a target-guided multimodal combined image retrieval method, comprising the following steps:

[0008] Extract attribute features of reference image, modified text and target image from local and global perspectives;

[0009] Based on the attribute features of the reference image, modified text and target image and the multimodal fusion module guided by the target-query relationship, multimodal feature fusion is performed to obtain the fused multimodal features;

[0010] In the target-query relationship-guided multimodal fusion module, two branches are designed. In the target-based teacher branch, the consistency relationship between the target image and the multimodal query is introduced as a guide to model the conflicting relationship between the reference image and the modified text in the multimodal query. In the target-free student branch, guidance information is indirectly obtained by learning from the teacher branch.

[0011] The fused multimodal features and target image attribute features are optimized through metric learning to obtain the basis for detection and retrieve the target image.

[0012] A second aspect of the present invention provides a target-guided multimodal combined image retrieval system, comprising:

[0013] An attribute feature extraction module is used to extract attribute features of the reference image, the modified text, and the target image from local and global perspectives;

[0014] A multimodal fusion module is used to perform multimodal feature fusion based on the attribute features of the reference image, the modified text and the target image and the target-query relationship to obtain fused multimodal features;

[0015] In the target-query relationship-guided multimodal fusion module, two branches are designed. In the target-based teacher branch, the consistency relationship between the target image and the multimodal query is introduced as a guide to model the conflicting relationship between the reference image and the modified text in the multimodal query. In the target-free student branch, guidance information is indirectly obtained by learning from the teacher branch.

[0016] The metric learning module is used to perform metric learning optimization on the fused multimodal features and target image attribute features to obtain the basis for detection and retrieve the target image.

[0017] A third aspect of the present invention provides a computer-readable storage medium.

[0018] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the target-guided multimodal combined image retrieval method as described above.

[0019] A fourth aspect of the present invention provides a computer device.

[0020] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the target-guided multimodal combined image retrieval method described above are implemented.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] 1. The present invention introduces the consistency relationship between the target image and the multimodal query into multimodal fusion for the first time, which can effectively model the conflicting relationships in multimodal queries.

[0023] 2. This paper is the first to introduce the similarity relationship between the target image and other images in the training batch into the batch-based image classification loss, regularize the loss function, and model the similarity ranking of images other than the target image during the metric learning process.

[0024] 3. The attribute feature extraction module designed by the present invention can uniformly extract attribute features from the reference image, modified text, and target image in this task from both local and global perspectives. Experimental results on public datasets demonstrate the effectiveness of the present invention.

[0025] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0027] Figure 1 This is a flowchart of a target-guided multimodal combined image retrieval method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0029] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0030] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0031] In order to solve the problems mentioned in the background technology, the present invention proposes a target-guided multimodal combined image retrieval method. The retrieval model adopted includes three key modules: an attribute feature extraction module, a target-query relationship-guided multimodal fusion module, and a target similarity distribution-guided metric learning module.

[0032] The attribute feature extraction module aims to extract unified attribute features for the reference image, modified text, and target image from both local and global perspectives. To ensure the independence of the extracted unified attribute features without attribute label supervision, the present invention designs an orthogonal regularization method to impose constraints.

[0033] The target-query relationship-guided multimodal fusion module consists of two branches: a target-based teacher branch and an untargeted student branch. The target-based teacher branch uses the consistency relationship between the target image and the multimodal query as a guide to model the conflicting relationships between the reference image and the modified text in the multimodal query. The untargeted student branch indirectly acquires this guidance by learning from the teacher branch.

[0034] In the metric learning module guided by target similarity distribution, the present invention aims to improve the traditional classification loss function based on training batches by introducing the similarity between the target image and other images in the training batch as a regularization constraint to guide the similarity ranking modeling of other images except the target image during the metric learning process.

[0035] Example 1

[0036] like Figure 1 As shown, this embodiment provides a target-guided multimodal combined image retrieval method, comprising the following steps:

[0037] Step 1: Use the attribute feature extraction module to extract unified attribute features of the reference image, modified text, and target image;

[0038] Step 2: The unified attribute features extracted in step 1 are fed into the target-query relationship guided multimodal fusion module for multimodal feature fusion.

[0039] Step 3: The multimodal query features and target image features obtained in step 2 are fed into the metric learning module guided by the target similarity distribution for metric learning optimization.

[0040] Wherein, the step 1 specifically includes the following steps:

[0041] Step 101: Using the multimodal pre-trained model CLIP as a feature extractor, perform feature extraction on the reference image, the modified text, and the target image. Since all three use the same process for extracting unified attribute features, feature extraction on the reference image is used as an example.

[0042] Assume that the global feature of the reference image obtained by CLIP is Add P learnable masks to the global feature to extract P global attribute features. The formula is:

[0043]

[0044] in, is the i-th mask, is the i-th global attribute feature of the reference image.

[0045] Step 102: Assume that the local features of the reference image obtained by CLIP are First, use the fully connected layer to map it to the same dimension as the global feature, and get L is the number of local features obtained after CLIP of the reference image, D′ is the local feature dimension, and D is the global feature dimension.

[0046] Attribute features are also extracted from local features. Assuming there are Q local features, the present invention designs Q feature aggregation functions, where the jth function The calculation formula is,

[0047]

[0048] in, is the jth local attribute feature extracted, Conv j (·) is a 1×1 convolutional network that aggregates the j-th feature, and σ is the Sigmoid activation function. is the weight vector for aggregating local features of the image, is matrix multiplication.

[0049] Step 103: Based on the global attribute features and local attribute features obtained in steps 101 and 102, the two are cascaded to obtain the final attribute features of the reference image. Where K = P + Q. Using the same method, attribute features can also be extracted for the modified text and target image, which are expressed as and Since the network used to extract attribute features is parameter-shared, their attribute features are unified, that is, attribute features at the same position should represent the same attribute meaning.

[0050] Step 104: Since there is no clear attribute label for supervision when extracting unified attribute features, the present invention introduces orthogonal regularization to ensure that the extracted attribute features are independent of each other. The orthogonal regularization constraint formula is:

[0051]

[0052] in, is the unit matrix, is the Frobenius-2 norm.

[0053] Step 2 specifically includes:

[0054] Step 201: In multimodal fusion, the present invention aims to guide multimodal fusion by utilizing the target-query relationship. Since the target image information cannot be obtained in the actual testing phase, a target-based teacher branch and a target-free student branch are designed.

[0055] Step 202: In the non-target student branch, multimodal fusion is performed using only the reference image and the modified text. First, the retention mask and the replacement mask are calculated using the formula:

[0056]

[0057] Among them, MLP s (·) is the multilayer perceptron network that extracts the mask in the non-target student branch, To preserve the mask, it represents the part of the consistent attribute features of the reference image that needs to be preserved. The replacement mask represents the part of the consistency attribute feature of the reference image that needs to be replaced by the consistency attribute feature of the modified text. Finally, the multimodal fusion feature is obtained in the following way:

[0058]

[0059] Step 203: In the target-based teacher branch, the target image information is introduced to obtain the retention mask and the replacement mask, and the final multimodal fusion feature is obtained through the same mask weighted addition method. The specific calculation formula is:

[0060]

[0061] in, and Here, we add an additional constraint that the sum of the two masks is 1 to promote the learning of the mask. The constraint function is: MLP t1 (·) is the multi-layer perceptron network that extracts the retention mask in the teacher branch, MLP t2 (·) is a multilayer perceptron network that extracts the replacement mask.

[0062]

[0063] Step 204: In order to enable the non-targeted student branch to learn the excellent multimodal fusion ability of the target-based teacher branch, the following constraints are added to the retention mask and replacement mask of the two branches:

[0064]

[0065] The step 3 specifically includes:

[0066] Step 301: In the target similarity distribution guided metric learning module, firstly, the multimodal fusion features and target image features obtained from the two branches in step (2) are constrained. The loss function used is a commonly used classification loss function based on training batches.

[0067] The goal-based teacher branch is only used during the model training phase. Therefore, a late-stage fusion approach is used to fuse the unified attribute features of multimodal queries to generate the vectors used for retrieval. This approach has the advantage of good results, but also low efficiency.

[0068] For the untargeted student branch, training is required during the training phase and inference is performed during the testing phase. Therefore, an early fusion approach is adopted for this part to ensure the efficiency of model retrieval.

[0069] The loss function for constraining these two branches is:

[0070]

[0071] Where B is the batch size, s(·,·) is the cosine similarity function, τ is the temperature coefficient, and K is the number of attribute features. E ti [k] and E tj [k] is the k-th row vector in the three matrices, that is, the k-th attribute. ψ ti and ψ tj Respectively for E ti and E tj The vector obtained by average pooling.

[0072] Step 302: In addition, the similarity distribution between the target image and other images in the training batch is used to guide the similarity ranking modeling between the multimodal query and other images in the training batch.

[0073] make represents the similarity distribution between the i-th target image and other images in the training batch, where is the normalized similarity score between the i-th target image and the j-th image in the training batch.

[0074] The calculation method is:

[0075]

[0076] Among them, E tb [k] is the kth attribute feature of the bth image in the training batch, E tj [k] is the k-th attribute feature of the j-th image in the training batch.

[0077] Step 303: Corresponding to step 302, the similarity distribution between the multimodal query features in the non-target student branch and the images in the training batch is calculated, and promoted to learn with the distribution calculated in step 302.

[0078] make represents the similarity distribution between the i-th multimodal query and other images in the training batch, where is the normalized similarity score between the i-th multimodal query and the j-th image in the training batch.

[0079] The calculation method is:

[0080]

[0081] Where, ψ tb is the feature obtained by average pooling of the b-th image in the training batch.

[0082] Step 304: Use the similarity distribution obtained in step 302 to guide the learning of the similarity distribution obtained in step 303.

[0083] This embodiment uses the Kullback Leibler divergence for constraint, and the formula is:

[0084]

[0085] Where D KL (·) denotes the Kullback-Leibler divergence constraint.

[0086] Finally, the model optimizes the parameters of the three components, and the optimization goal is,

[0087]

[0088] Among them, Θ is the parameter to be optimized in the TG-CIR model, and λ, η, μ, ν and κ are balance hyperparameters.

[0089] Example 2

[0090] This embodiment provides a target-guided multimodal combined image retrieval system, including:

[0091] An attribute feature extraction module is used to extract attribute features of the reference image, the modified text, and the target image from local and global perspectives;

[0092] A multimodal fusion module is used to perform multimodal feature fusion based on the attribute features of the reference image, the modified text and the target image and the target-query relationship to obtain fused multimodal features;

[0093] In the target-query relationship-guided multimodal fusion module, two branches are designed. In the target-based teacher branch, the consistency relationship between the target image and the multimodal query is introduced as a guide to model the conflicting relationship between the reference image and the modified text in the multimodal query. In the target-free student branch, guidance information is indirectly obtained by learning from the teacher branch.

[0094] The metric learning module is used to perform metric learning optimization on the fused multimodal features and target image attribute features to obtain the basis for detection and retrieve the target image.

[0095] Example 3

[0096] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the program implements the steps in the target-guided multi-modal combined image retrieval method as described above.

[0097] Example 4

[0098] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the target-guided multimodal combined image retrieval method described above are implemented.

[0099] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0100] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0101] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0102] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0103] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0104] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A target-guided multimodal combined image retrieval method, characterized in that: The steps include: Extract attribute features of reference image, modified text and target image from local and global perspectives; Based on the attribute features of the reference image, modified text and target image and the multimodal fusion module guided by the target-query relationship, multimodal feature fusion is performed to obtain the fused multimodal features; In the target-query relationship-guided multimodal fusion module, two branches are designed. In the target-based teacher branch, the consistency relationship between the target image and the multimodal query is introduced as a guide to model the conflicting relationship between the reference image and the modified text in the multimodal query. In the target-free student branch, guidance information is indirectly obtained by learning from the teacher branch. The fused multimodal features and target image attribute features are optimized through metric learning to obtain the basis for detection and retrieve the target image; The multimodal fusion module based on the attribute features of the reference image, the modified text and the target image and the target-query relationship performs multimodal feature fusion to obtain fused multimodal features, specifically including: Designed a goal-based teacher branch and a goal-free student branch; In the non-target student branch, only the reference image and the modified text are used for multimodal fusion, and a first retention mask and a first replacement mask are obtained according to the attribute features of the reference image and the modified text; a first multimodal fusion feature is obtained based on the first retention mask and the first replacement mask; In the target-based teacher branch, target image information is introduced to obtain a second retention mask and a second replacement mask, and a second multimodal fusion feature is obtained through the second retention mask and the second replacement mask; The first retention mask represents the portion of the consistent attribute features of the reference image that needs to be retained, and the first replacement mask represents the portion of the consistent attribute features of the reference image that needs to be replaced by the consistent attribute features of the modified text; In the goal-based teacher branch, the specific calculation formula is: In the non-target student branch, the specific calculation formula is: ， Add the following constraints to the retain mask and replacement mask of the two branches: ， in, and are the retention mask and replacement mask in the teacher branch, respectively. It is a multi-layer perceptron network that extracts the retention mask in the teacher branch is a multilayer perceptron network that extracts the replacement mask, and To modify the attribute features of text and target image, is the attribute feature of the reference image, is the Sigmoid activation function, A multilayer perceptron network that extracts masks from the untargeted student branch, and Keep mask and replace mask for student branch.

2. The target-guided multimodal combined image retrieval method according to claim 1, wherein: The extracting of attribute features of the reference image, the modified text, and the target image from local and global perspectives specifically includes: Extract corresponding global features and local features through multimodal pre-training models; Add a learnable mask to the global features to extract the corresponding global attribute features; The fully connected layer is used to map the local features to the same dimension as the global features, and the attribute features are also extracted from the local features to obtain the local attribute features; The global attribute features and local attribute features are cascaded to obtain the final attribute features.

3. The target-guided multimodal combined image retrieval method according to claim 1, wherein: When extracting unified attribute features of the reference image, modified text, and target image from local and global perspectives, an orthogonal regularization method is introduced to constrain them to ensure that the extracted unified attribute features are independent of each other.

4. The target-guided multimodal combined image retrieval method according to claim 1, wherein: When performing metric learning on the fused multimodal features and target image attribute features, the traditional classification loss function based on training batches is improved by introducing the similarity between the target image and other images in the training batch as a regularization constraint to guide the learning of the metric function in the metric learning process.

5. The target-guided multimodal combined image retrieval method according to claim 4, characterized in that: The method introduces the similarity between the target image and other images in the training batch as a regularization constraint to guide the learning of the metric function in the metric learning process, specifically including: Based on the target teacher branch, calculate the target image and the The normalized similarity score of the image is obtained based on the normalized similarity score. The similarity distribution between the target image and other images in the training batch; In the non-target student branch, calculate the multimodal query and the training batch The normalized similarity score of the image is obtained based on the normalized similarity score. Similarity distribution between a multimodal query and other images in the training batch; The first similarity distribution is used to guide the second similarity distribution.

6. A target-guided multimodal combined image retrieval system, using a target-guided multimodal combined image retrieval method according to any one of claims 1 to 5, characterized in that: include: An attribute feature extraction module is used to extract attribute features of the reference image, the modified text, and the target image from local and global perspectives; A multimodal fusion module is used to perform multimodal feature fusion based on the attribute features of the reference image, the modified text and the target image and the target-query relationship to obtain fused multimodal features; In the target-query relationship-guided multimodal fusion module, two branches are designed. In the target-based teacher branch, the consistency relationship between the target image and the multimodal query is introduced as a guide to model the conflicting relationship between the reference image and the modified text in the multimodal query. In the target-free student branch, guidance information is indirectly obtained by learning from the teacher branch. The metric learning module is used to perform metric learning optimization on the fused multimodal features and target image attribute features to obtain the basis for detection and retrieve the target image.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the target-guided multimodal combined image retrieval method according to any one of claims 1 to 5 are implemented.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps in the target-guided multimodal combined image retrieval method according to any one of claims 1 to 5 are implemented.

Images