An information retrieval method and system based on combined multi-modal query
Patent Information
- Application Number
- CN202410377835.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-29
- Publication Date
- 2026-11-03
- Estimated Expiration
- 2044-03-29
AI Technical Summary
Existing multimodal combined image retrieval methods based on vision-language pre-trained large models are prone to deviating from the original pre-trained feature space during feature fusion, resulting in retrieval results that deviate from the target image features.
Multimodal fusion is shifted from the feature level to the raw data level. By linearly weighting the fusion of features from reference images and modified text, a large vision-language pre-trained model is used to extract image and text features, and multimodal fusion is performed at the raw data level. Feature balancing is achieved by combining a multilayer perceptron and a sigmoid activation function.
It achieves superior multimodal combined image retrieval performance, effectively integrating diverse user modification needs. The model structure is simple and effective, and by utilizing the multimodal feature extraction and cross-modal retrieval capabilities of the VLP model, it improves the accuracy and flexibility of retrieval.
Smart Images
Figure CN118193769B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image retrieval technology, specifically relating to an information retrieval method and system based on combined multimodal query, and more particularly to a combined image retrieval method, system, medium, and device based on multimodal fusion at the raw data level. Background Technology
[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.
[0003] Compared to traditional image retrieval tasks based on single-modal queries, multimodal combined image retrieval tasks offer a more flexible retrieval method. It allows users to combine multiple modal query conditions for image searching. In this case, the reference image is used to express the user's overall search intent, while supplementary modified text is used to indicate local adjustments the user wishes to make to certain attributes of the reference image.
[0004] In recent years, thanks to the excellent multimodal feature extraction and cross-modal retrieval capabilities of the Vision-Language Pre-trained model (VLP), multimodal combined image retrieval methods based on VLP models have made some progress. Existing VLP-based methods often introduce complex nonlinear multimodal fusion modules after extracting reference image features and modifying text features, ultimately using the fused features to retrieve the target image. This approach, by introducing additional nonlinear operations after the features obtained from the VLP model, causes the fused features to deviate from the original pre-trained feature space, thus resulting in a deviation from the target image features. Summary of the Invention
[0005] To address the aforementioned problems, this invention proposes an information retrieval method and system based on combined multimodal queries. This invention shifts multimodal fusion from the feature level of existing methods to the raw data level, enabling superior multimodal combined image retrieval results. Image retrieval can be performed based on multimodal queries, including reference images and modified text descriptions of the reference images, to retrieve images that meet the requirements.
[0006] According to some embodiments, the present invention adopts the following technical solution:
[0007] An information retrieval method based on combined multimodal queries includes the following steps:
[0008] Obtain a reference image and modified text, extract the image description from the reference image, and concatenate the extracted image description and modified text to obtain text-based query conditions;
[0009] The modified text is analyzed to extract key descriptions of the target image. These key descriptions are then written into a reference image to obtain image-oriented query conditions.
[0010] The two query conditions are used to extract features using a large vision-language pre-trained model to obtain corresponding text features and image features. These features are then fused using linear weighting. Based on the fused combined query features, the target image is retrieved.
[0011] As an alternative implementation, the process of extracting image descriptions from a reference image includes: using a multimodal pre-trained model BL IP-2 to extract image descriptions from the reference image, thereby obtaining the extracted image descriptions.
[0012] As an alternative implementation, the process of concatenating the extracted image description and modified text includes combining the image description and modified text to form a query condition that satisfies the image description, but where a certain piece of information satisfies the plain text form of the modified text.
[0013] As an alternative implementation method, the process of analyzing modified text includes: analyzing the modified text using a large language model, extracting key descriptions of the target image using cue learning methods, and segmenting the extracted descriptions.
[0014] As an alternative implementation, the specific process of writing the key description into the reference image includes: the obtained target image key description words are directly written onto the reference image using the cv2.putText() function of the OpenCV library to obtain query conditions in pure image form.
[0015] As an alternative implementation, the process of feature extraction using a large-scale vision-language pre-trained model includes: using the CLIP model in the large-scale vision-language pre-trained model to extract features from the plain text query conditions q. t And query conditions in pure image form q v Perform feature extraction;
[0016]
[0017] in, and For the corresponding text features and image features, D represents the dimension of the feature.
[0018] As an alternative implementation method, the fusion process includes fusing the extracted text features and image features using a linear weighting operation;
[0019]
[0020] in, λ is the final combined query feature, λ is the balance parameter for balancing text features and image features, [·∥·] is the concatenation operation, MLP is a multilayer perceptron, and σ is the Sigmoid activation function.
[0021] As an alternative implementation method, the process of retrieving target images based on the fused combined query features includes: extracting features from the target image using the same CLIP model, comparing the combined query features with the corresponding target image features, and determining the target image with the smallest difference between the two features as the final retrieval result.
[0022] An information retrieval system based on combined multimodal queries includes:
[0023] The text-oriented multimodal fusion module is configured to acquire a reference image and modified text, extract image descriptions from the reference image, and concatenate the extracted image descriptions and modified text to obtain text-oriented query conditions.
[0024] The image-oriented multimodal fusion module is configured to analyze the modified text, extract key descriptions about the target image, write the key descriptions into a reference image, and obtain image-oriented query conditions.
[0025] The target retrieval module based on linear weighted fusion is configured to extract features from two obtained query conditions using a vision-language pre-trained large model to obtain corresponding text features and image features, perform linear weighted fusion, and retrieve the target image based on the fused combined query features.
[0026] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the steps in the above method.
[0027] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the steps in the method described above.
[0028] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0029] This invention attempts to perform multimodal fusion operations at the raw data level in image retrieval tasks. The multimodal fusion method designed in this invention, which is geared towards both images and text, can fully utilize the excellent multimodal feature extraction and cross-modal retrieval capabilities of the VLP model.
[0030] This invention effectively integrates the two proposed multimodal fusion methods, thereby effectively addressing diverse user modification needs. The specific implementation requires only a model comprising a text encoder, an image encoder, and a multilayer perceptron; the model structure is simple and effective.
[0031] This invention utilizes publicly available datasets for experiments, and the results demonstrate the effectiveness of the proposed method. The experimental results also reveal that the VLP model's image encoder possesses excellent Optical Character Recognition (OCR) capabilities. This can inspire other multimodal fusion work to design model schemes from a new perspective.
[0032] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0033] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0034] Figure 1 This is a schematic diagram of the model in this embodiment. Detailed Implementation
[0035] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0036] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0037] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0038] Where there is no conflict, the embodiments and features described in this application may be combined with each other.
[0039] Example 1
[0040] An information retrieval system based on combined multimodal queries, implemented using the DQU-CI R model, such as... Figure 1As shown, it includes:
[0041] The text-oriented multimodal fusion module is configured to acquire a reference image and modified text, extract image descriptions from the reference image, and concatenate the extracted image descriptions and modified text to obtain text-oriented query conditions.
[0042] The image-oriented multimodal fusion module is configured to analyze the modified text, extract key descriptions about the target image, write the key descriptions into a reference image, and obtain image-oriented query conditions.
[0043] The target retrieval module is configured to extract features from the two obtained query conditions using a large vision-language pre-trained model, obtain corresponding text features and image features, fuse them, and perform target image retrieval based on the fused combined query features.
[0044] Example 2
[0045] An information retrieval method based on combined multimodal queries includes the following steps:
[0046] (1) Obtain the reference image and modified text, extract the image description from the reference image, and concatenate the extracted image description and modified text to obtain the text-oriented query conditions;
[0047] (2) Analyze the modified text, extract the key descriptions of the target image, and write the key descriptions into the reference image to obtain image-oriented query conditions;
[0048] (3) The two query conditions are used to extract features using a vision-language pre-trained large model to obtain the corresponding text features and image features. These features are then fused together, and the target image is retrieved based on the fused combined query features.
[0049] The dataset used in this embodiment is in triplet form, represented as follows: Where, x r Represents the reference image, t m This indicates text modification, and x t This represents the corresponding target image. N is the number of triples.
[0050] Specifically, the specific steps of step (1) include:
[0051] (1-1) The multimodal pre-trained model BL IP-2 is used to extract image descriptions from the reference image. The formula is as follows:
[0052] t r =BLIP-2(x r ),
[0053] Among them, tr For the reference image x r The extracted image description.
[0054] (1-2) Image description t of the obtained reference image r With modified text t m By splicing, we get "t" r , but t m The query condition q is in plain text form. t The query criteria include necessary descriptive information about the reference image, as well as sufficient information about the modifications to the text, allowing for direct feature extraction using the VLP model's text encoder. Because this operation fully preserves the modified text, it is more suitable for complex modification scenarios.
[0055] Step (2) includes the following specific steps:
[0056] (2-1) The modified text is analyzed using a Large Language Model (LLM), and key descriptions of the target image are extracted using a cue learning method. The cue learning template is designed as follows: "Given text describes the differences between two images. Please analyze it, extract key descriptions of the target image, and segment the extracted descriptions with commas. Ensure redundancy is eliminated and spelling errors are corrected." This operation fully utilizes the powerful natural language processing capabilities of LLM to directly extract key descriptions of the target image.
[0057] (2-2) The target image keywords obtained in step (2-1) are directly written to the reference image t using the cv2.putText() function of the OpenCV library. r The above yields the query condition q in pure image form. v This method preserves the visual information of the reference image sufficiently, and the key descriptions of the target image in the modified text are directly written onto the reference image, allowing for subsequent feature extraction using the VLP model's image encoder. This method is more suitable for situations where the modification to the reference image is minor.
[0058] Step (3) includes the following specific steps:
[0059] (3-1) Through steps (1) and (2), the q values of the multimodal fusion results at the original data level were obtained respectively. t and q v Among them, q t Because it fully preserves the modified text information, it is more suitable for modifying complex situations. And q vIt fully preserves the visual information of the reference image, making it more suitable for simple modifications. First, the CLIP model in the VLP model is used to modify q. t and q v Feature extraction is performed using the following formula:
[0060]
[0061] in, and For the corresponding text features and image features, D represents the dimension of the feature.
[0062] (3-2) To ensure the model can flexibly respond to diverse user modification needs, a linear weighting operation is designed to fuse the extracted text and image features. The formula is as follows:
[0063]
[0064] in, This represents the final combined query feature, where λ is the balancing parameter between text and image features, [·∥·] is the concatenation operation, MLP is a multilayer perceptron, and σ is the sigmoid activation function. It's worth noting that the operations introduced here are linear, therefore they will not cause the extracted combined query features to deviate from the original pre-trained feature space.
[0065] (3-3) Using the same CLIP model, feature extraction is performed on the target image. The formula for this process is:
[0066] f t =CLIP image (x t ),
[0067] in, Features of the target image.
[0068] (3-4) Compare the target image features with the final combined query features. If the difference between the two features is less than the set value or the minimum, then the target image is the final query result.
[0069] In some embodiments, each step of the above method relies on the combined image retrieval system provided in Embodiment 1, that is, based on the DQU-CIR model.
[0070] Step (1) is implemented by a text-oriented multimodal fusion module;
[0071] Step (2) is implemented by the image-oriented multimodal fusion module;
[0072] Step (3) is implemented by the target retrieval module based on linear weighted fusion.
[0073] In the optimization process of the DQU-CIR model, step (3) also includes:
[0074] By using a contrastive learning loss function, the combined query features f are made more efficient. q With the corresponding target image features f t The formula for the loss function is as follows:
[0075]
[0076] Where, the subscript i represents the i-th triplet sample in the mini-batch data, B is the number of samples in the mini-batch data, cos(·,·) represents the cosine similarity calculation function, and τ represents the temperature parameter.
[0077] Finally, the objective function for optimizing DQU-CIR is expressed as follows:
[0078]
[0079] Where Θ represents the DQU-CIR network parameters to be optimized.
[0080] Example 3
[0081] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the steps in the method provided in Embodiment 2.
[0082] Example 4
[0083] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the steps in the method provided in Embodiment 2.
[0084] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0085] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0086] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0087] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0088] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made by those skilled in the art without creative effort within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. An information retrieval method based on combined multimodal queries, characterized in that, Includes the following steps: Obtain a reference image and modified text, extract the image description from the reference image, and concatenate the extracted image description and modified text to obtain text-based query conditions; The modified text is analyzed to extract key descriptions of the target image. These key descriptions are then written into a reference image to obtain image-oriented query conditions. The two query conditions are used to extract features using a large vision-language pre-trained model to obtain corresponding text features and image features. These features are then fused using linear weighting. Based on the fused combined query features, the target image is retrieved. The process of extracting image descriptions from a reference image includes: using the multimodal pre-trained model BLIP-2 to extract image descriptions from the reference image, and obtaining the extracted image descriptions from the reference image; The specific process of writing the key description into the reference image includes: the obtained target image key description words are directly written onto the reference image using the cv2.putText() function of the OpenCV library to obtain query conditions in pure image form.
2. The information retrieval method based on combined multimodal query as described in claim 1, characterized in that, The process of concatenating the extracted image description and modified text includes, for example, the image description of the obtained reference image. With modified text By splicing them together, we get " ,but The query conditions are in plain text format.
3. The information retrieval method based on combined multimodal query as described in claim 1, characterized in that, The process of analyzing modified text includes: analyzing the modified text using a large language model, extracting key descriptions of the target image using cue learning methods, and segmenting the key descriptions.
4. The information retrieval method based on combined multimodal query as described in claim 1, characterized in that, The process of feature extraction using a large-scale vision-language pre-trained model includes: using the CLIP model within the large-scale vision-language pre-trained model to extract features from query conditions in plain text format. And query conditions in pure image form Perform feature extraction; in, and For the corresponding text features and image features, The dimension representing the feature.
5. The information retrieval method based on combined multimodal query as described in claim 4, characterized in that, The fusion process includes fusing the extracted text features and image features using a linear weighting operation; in, It is the final combined query feature. To balance the balance parameters between text features and image features, For splicing operations, It is a multilayer perceptron. This is the Sigmoid activation function.
6. The information retrieval method based on combined multimodal query as described in claim 1, characterized in that, The process of target image retrieval based on the fused combined query features includes: extracting features from the target image using the same CLIP model, comparing the combined query features with the corresponding target image features, and determining the target image with the smallest difference between the two features as the final retrieval result.
7. An information retrieval system based on combined multimodal queries, characterized in that, include: The text-oriented multimodal fusion module is configured to acquire a reference image and modified text, extract image descriptions from the reference image, and concatenate the extracted image descriptions and modified text to obtain text-oriented query conditions. The image-oriented multimodal fusion module is configured to analyze the modified text, extract key descriptions about the target image, write the key descriptions into a reference image, and obtain image-oriented query conditions. The target retrieval module based on linear weighted fusion is configured to extract features from the two obtained query conditions using a vision-language pre-trained large model to obtain corresponding text features and image features, perform linear weighted fusion, and retrieve the target image based on the fused combined query features. The process of extracting image descriptions from a reference image includes: using the multimodal pre-trained model BLIP-2 to extract image descriptions from the reference image, and obtaining the extracted image descriptions from the reference image; The specific process of writing the key description into the reference image includes: the obtained target image key description words are directly written onto the reference image using the cv2.putText() function of the OpenCV library to obtain query conditions in pure image form.
8. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the steps of the method according to any one of claims 1-6.