Multi-modal image retrieval system and method based on multi-granularity keyword analysis
By combining multi-granularity keyword parsing and multimodal models, the problem of insufficient granularity processing in multimodal image retrieval is solved, achieving more efficient and accurate image retrieval, and improving retrieval accuracy and user satisfaction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NAT UNIV OF DEFENSE TECH
- Filing Date
- 2025-07-01
- Publication Date
- 2026-06-09
AI Technical Summary
Existing multimodal image retrieval technologies suffer from limitations in understanding complex semantics, such as insufficient keyword granularity processing, and limited ability to comprehend dependencies and spatial relationships between keywords, when handling query requests with different granularities of information. This results in low retrieval accuracy and user satisfaction.
A multimodal image retrieval system and method based on multi-granularity keyword parsing is adopted. The language parsing module performs multi-granularity analysis on the query text input by the user, identifies elements of different granularities and decomposes them into multi-granularity keywords. The CLIP model and LLM model are combined to extract feature vectors and perform comprehensive semantic matching. The CLIP model is used to filter candidate images and switch to global retrieval when necessary to ensure the reliability of the results.
It improves the accuracy and efficiency of image retrieval, better understands users' detailed query needs, returns images that meet users' actual needs, and ensures the accuracy and reliability of search results.
Smart Images
Figure CN122173670A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image retrieval technology, and in particular to a multimodal image retrieval system and method based on multi-granularity keyword parsing. Background Technology
[0002] With the widespread adoption of the internet and mobile devices, digital image data is exploding. How to quickly and accurately retrieve the content users need from this massive amount of image data has become a major challenge in the field of information technology. Traditional retrieval methods based on text tags or single modalities (such as image content only) are often insufficient to meet users' increasingly complex and refined search needs.
[0003] To compensate for the limitations of single-modal information, multimodal image retrieval has emerged. This technology aims to fuse the visual content information of an image with its associated textual description, tags, or other modal information, achieving image retrieval that better aligns with human cognitive habits and semantic needs through cross-modal understanding and matching. For example, users can input a natural language description to find images containing specific scenes, objects, actions, or emotions. In recent years, deep learning technologies, especially pre-trained models such as Transformer and CLIP (Contrastive Language-Image Pre-training), have greatly promoted the development of multimodal understanding and representation learning, enabling models to better learn the potential relationships between text and images.
[0004] However, user-input search requests (keywords or descriptions) and the information contained in the images themselves often have different granularities. Currently, although existing multimodal image retrieval technologies have made some progress, they still face challenges in handling queries with varying granularities, such as insufficient keyword granularity processing and limited understanding of complex semantics including dependencies and spatial relationships between keywords. Therefore, the accuracy and user satisfaction of multimodal image retrieval need further improvement. Summary of the Invention
[0005] Therefore, it is necessary to provide a multimodal image retrieval system and method based on multi-granularity keyword parsing to address the aforementioned technical problems.
[0006] A multimodal image retrieval system based on multi-granularity keyword parsing, the system comprising a language parsing module, a multimodal retrieval module, and a semantic rearrangement module connected in sequence;
[0007] The language parsing module is used to perform multi-granularity analysis on the query text input by the user through a large model, identify elements of different granularities in the query text, and decompose each element into multi-granularity keywords;
[0008] The multimodal retrieval module is used to take the multi-granular keywords output by the language parsing module as semantic labels and input them into the CLIP model for feature vector extraction. By comparing the cosine similarity between the feature vector of the semantic label and the feature vector of each image stored in the database, several highly relevant images corresponding to each semantic label are selected.
[0009] The semantic reordering module takes all highly relevant images output by the multimodal retrieval module as candidate images, inputs the candidate images and the complete query text into the LLM (Large Language Model) for comprehensive semantic matching and reordering, and outputs the most relevant image of the query text. It further calculates and judges whether the similarity between the feature vector of the most relevant image and the feature vector of the query text is lower than a preset threshold. If it is lower, the CLIP model is called again to perform a global search on all images in the database and returns the image with the highest similarity to the feature vector of the query text as the target image; otherwise, the most relevant image output by the LLM model is directly used as the target image.
[0010] In one embodiment, before performing multi-granularity analysis on the user-input query text through the large model, the language parsing module is also used to: call the translation API (Application Programming Interface) to translate the user-input query text into the language adapted for processing by the large model, and input the translated query text into the large model for multi-granularity analysis.
[0011] In one embodiment, within the language parsing module, the large model identifies elements of different granularities, including objects, attributes, and spatial relationships in the query text.
[0012] In one embodiment, before comparing the cosine similarity between the feature vectors of semantic tags and the feature vectors of each image stored in the database, the multimodal retrieval module is further configured to: call the CLIP model to extract the feature vectors of each image in the database, and store the feature vectors of each image in the database.
[0013] In one embodiment, the multimodal retrieval module is further configured to: create and start an independent thread for each semantic tag and each image, and in each thread, call the CLIP model to extract the corresponding feature vector.
[0014] In one embodiment, the semantic reordering module calculates and determines whether the similarity between the feature vector of the most relevant image and the feature vector of the query text is lower than a preset threshold, including:
[0015] The CLIP model is called to extract the feature vector of the query text, and it is determined whether the cosine similarity between the feature vector of the query text and the feature vector of the most relevant image stored in the database is lower than a preset threshold.
[0016] In one embodiment, the semantic reordering module re-invokes the CLIP model to perform a global search on all images in the database and returns the image with the highest similarity to the feature vector of the query text as the target image, including:
[0017] The query text is input into the LLM model again, and the LLM model is used to extract the text describing spatial relationships in the query text and extract the corresponding feature vectors.
[0018] The feature vectors of the text describing spatial relationships are superimposed with the feature vectors of the query text to generate text feature vectors with spatial relationship semantic sharpening. The CLIP model is then called again for global retrieval. The CLIP model is used to compare the cosine similarity between the text feature vectors with spatial relationship semantic sharpening and the feature vectors of each image stored in the database. The image with the highest cosine similarity to the text feature vectors with spatial relationship semantic sharpening is selected as the target image.
[0019] A multimodal image retrieval method based on multi-granularity keyword parsing, the method comprising:
[0020] By using a large model to perform multi-granularity analysis on the query text input by users, elements of different granularities in the query text are identified and each element is decomposed into multi-granularity keywords.
[0021] Multi-granular keywords are used as semantic labels and input into the CLIP model for feature vector extraction. By comparing the cosine similarity between the feature vectors of the semantic labels and the feature vectors of each image stored in the database, several highly relevant images corresponding to each semantic label are selected.
[0022] All highly relevant images are selected as candidate images. The candidate images are then input into the LLM model along with the complete query text for comprehensive semantic matching and reordering. The most relevant image for the query text is output. The similarity between the feature vector of the most relevant image and the feature vector of the query text is calculated and determined to be lower than a preset threshold. If it is lower, the CLIP model is called again to perform a global search on all images in the database and the image with the highest similarity to the feature vector of the query text is returned as the target image. Otherwise, the most relevant image output by the LLM model is directly used as the target image.
[0023] In one embodiment, the method further includes:
[0024] The translation API is called to translate the user-input query text into the language that the large model is adapted to, and the translated query text is then input into the large model for multi-granularity analysis.
[0025] The CLIP model is called to extract the feature vector of each image in the database, and the feature vector of each image is stored in the database.
[0026] The aforementioned multimodal image retrieval system and method based on multi-granularity keyword parsing first extracts multi-granularity keywords from the query text through multi-granularity analysis. This fully leverages information at different granularities within the query text, facilitating subsequent model accurate understanding of the user's detailed query needs. Then, the CLIP model filters relevant images for each keyword from a large-scale image dataset and uses them as candidate images. A small number of candidate images are then input into an LLM model along with the user's initial complete query text for semantic reordering, outputting the most relevant image for the query text. This multi-stage image retrieval process avoids the inefficient practice of directly inputting a large number of images into an LLM model for relevance evaluation, significantly improving retrieval efficiency. Furthermore, compared to traditional image retrieval using a single CLIP model, the combination of the CLIP and LLM models facilitates deeper semantic understanding. The LLM model can understand spatial relationships, orientation descriptions, and complex contextual information that the CLIP model struggles to handle, thus returning images that better match the user's actual needs. Furthermore, after the LLM model outputs the most relevant image, a similarity threshold is used to determine whether the confidence level of the LLM output result is insufficient. In this case, the system automatically switches to CLIP global retrieval, which ensures the reliability of the final output retrieval results, ensures a high degree of matching between the target image and the query text, and improves the accuracy of multimodal image retrieval. Attached Figure Description
[0027] Figure 1 This is a schematic diagram illustrating the image retrieval process of a multimodal image retrieval system based on multi-granularity keyword parsing in one embodiment.
[0028] Figure 2 This is a flowchart illustrating a multimodal image retrieval method based on multi-granularity keyword parsing in one embodiment. Detailed Implementation
[0029] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0030] In one embodiment, a multimodal image retrieval system based on multi-granularity keyword parsing is provided. The system consists of a language parsing module, a multimodal retrieval module, and a semantic reordering module connected in sequence.
[0031] The language parsing module is used to perform multi-granularity analysis on the query text input by the user through a large model, identify elements of different granularities in the query text, and decompose each element into multi-granularity keywords.
[0032] Specifically, in the language parsing module, large-scale models such as Kimi and GPT can be used for multi-granularity analysis. This multi-granularity analysis extracts elements of different granularities from the query text and decomposes them into multi-granularity keywords. This fully leverages the different granularities of information within the query text. For example, for the query text "There is a red parasol on a sunny beach," multi-granularity analysis using a large-scale model can not only identify entity elements like "beach" and "parasol," but also focus on coarser-grained scene atmospheres like "sunny." Simultaneously, it can identify element information at different granular levels, such as "red" (an object attribute, a finer granularity), and associate the "red" attribute with "parasol" rather than "beach." This helps the subsequent model accurately understand the user's detailed query needs, avoiding overly broad search results or omissions of key details, thus improving the accuracy of image retrieval. Notably, multi-granularity keywords are not entirely independent words, but rather phrases composed of textual context, semantic relationships, or domain-specific knowledge.
[0033] The multimodal retrieval module takes the multi-granular keywords output by the language parsing module as semantic labels and inputs them into the CLIP model for feature vector extraction. By comparing the cosine similarity between the feature vectors of the semantic labels and the feature vectors of each image stored in the database, it selects several highly relevant images corresponding to each semantic label.
[0034] Among them, the multimodal retrieval module uses the CLIP model to filter out relevant images for each keyword from a large number of images and use them as candidate images. This helps to avoid the inefficient practice of directly inputting a large number of images into LLM to evaluate the relevance of the retrieval, and greatly improves the retrieval efficiency.
[0035] The semantic reordering module takes all highly relevant images output by the multimodal retrieval module as candidate images, inputs the candidate images and the complete query text into the LLM model for comprehensive semantic matching and reordering, and outputs the most relevant image of the query text. It further calculates and judges whether the similarity between the feature vector of the most relevant image and the feature vector of the query text is lower than a preset threshold. If it is lower, the CLIP model is called again to perform a global search on all images in the database and returns the image with the highest similarity to the feature vector of the query text as the target image; otherwise, the most relevant image output by the LLM model is directly used as the target image.
[0036] Specifically, the LLM model performs comprehensive semantic matching and re-ranking of candidate images and complete query text by: using the LLM model to map the query text and candidate images to the same semantic space, calculating their matching scores and re-ranking them, and outputting the image with the highest semantic matching score as the most relevant image for the query text.
[0037] Compared to traditional single CLIP model image retrieval, combining CLIP and LLM models facilitates deeper semantic understanding. LLM can understand spatial relationships, orientation descriptions, and complex contextual information that CLIP models struggle to handle, thus returning images that better match the user's actual needs. Furthermore, considering that in some cases, the multi-granular keywords obtained after word segmentation may lose some semantic information from the original input query text, causing CLIP models to return images that do not perfectly match the user's intent, and potentially failing to provide accurate candidate images for LLM, the semantic re-ranking module, after the LLM model outputs the most relevant image, uses a similarity threshold. When the similarity between the feature vector of the most relevant image and the feature vector of the query text is lower than a preset threshold, the confidence of the LLM output is considered insufficient. In this case, it automatically switches to CLIP for global retrieval, ensuring the reliability of the final output retrieval results and thus guaranteeing a high degree of matching between the target image and the query text, improving the accuracy of multimodal image retrieval.
[0038] In one embodiment, such as Figure 1 As shown, before performing multi-granularity analysis on the user-input query text using the large model, the language parsing module is also used to: call the translation API to translate the user-input query text into a language adapted for processing by the large model, and then input the translated query text into the large model for multi-granularity analysis. Specifically, the Baidu Translate API or other high-precision translation services (such as DeepL) can be used for translation.
[0039] In one embodiment, within the language parsing module, the large model identifies elements of different granularities, including objects, attributes, and spatial relationships within the query text. An object is defined as an entity or thing described in the text, typically the subject or object of a sentence, possessing independent existence. Attributes are defined as characteristics or states describing objects, including visual, functional, and property attributes, encompassing static attributes (such as color, shape, and material), dynamic attributes (such as behavior and state changes), and abstract attributes (such as emotion and value). Spatial relationships are defined as the relative positions, directions, or distances between objects, crucial for scene understanding. By identifying elements such as objects, attributes, and spatial relationships, the core intent of the query text can be fully and comprehensively analyzed, ensuring a deep understanding of every detail in subsequent processing.
[0040] In one embodiment, before comparing the cosine similarity between the feature vectors of semantic tags and the feature vectors of each image stored in the database, the multimodal retrieval module is further configured to: call the CLIP model to extract the feature vectors of each image in the database, and store the feature vectors of each image in the database for fast retrieval. In addition, the database also stores the shooting coordinates of each image, and the system provides the shooting coordinates of the target image to the user simultaneously with the output of the target image.
[0041] In one embodiment, the multimodal retrieval module is further configured to: create and launch an independent thread for each semantic tag and each image, and in each thread, call the CLIP model to extract the corresponding feature vector. The creation of independent threads enables the simultaneous processing of feature extraction from multiple semantic tags and images, avoiding serial blocking and ensuring that the CLIP model operates without conflict under multi-threaded calls, thus significantly improving the efficiency of image retrieval.
[0042] In one embodiment, the semantic reordering module calculates and determines whether the similarity between the feature vector of the most relevant image and the feature vector of the query text is lower than a preset threshold, including:
[0043] The CLIP model is called to extract the feature vector of the query text, and it is determined whether the cosine similarity between the feature vector of the query text and the feature vector of the most relevant image stored in the database is lower than a preset threshold.
[0044] For example, when a user queries and retrieves a street view image, before calling the CLIP model to extract feature vectors, the following steps are also included:
[0045] Obtain the Flickr30k dataset, which includes image and text annotations;
[0046] Images related to street views in the Flickr30k dataset were obtained by using text filtering methods (such as filtering for texts containing "street", "house", "park" and other street view related texts).
[0047] The Flickr30k dataset, after text filtering, was input into the pre-trained CLIP model for fine-tuning to improve the CLIP model's ability to understand street scene semantics and image features.
[0048] Finally, the fine-tuned CLIP model is used to extract feature vectors from semantic tags, images, and query text.
[0049] In one embodiment, the semantic reordering module re-invokes the CLIP model to perform a global search on all images in the database and returns the image with the highest similarity to the feature vector of the query text as the target image, including:
[0050] The query text is input into the LLM model again, and the LLM model is used to extract the text describing spatial relationships in the query text and extract the corresponding feature vectors.
[0051] The feature vectors of the text describing spatial relationships are superimposed with the feature vectors of the query text to generate text feature vectors with spatial relationship semantic sharpening. The CLIP model is then called again for global retrieval. The CLIP model is used to compare the cosine similarity between the text feature vectors with spatial relationship semantic sharpening and the feature vectors of each image stored in the database. The image with the highest cosine similarity to the text feature vectors with spatial relationship semantic sharpening is selected as the target image.
[0052] It is understandable that this application takes into account that even if global retrieval is performed using the CLIP model based solely on the original complete query text, spatial relationships may still be lost. Therefore, it further utilizes the deep semantic understanding capabilities of LLM to semantically sharpen the features of the original complete query text. That is, it extracts the feature vectors of the text describing spatial relationships in the query text and superimposes them with the original feature vectors to generate text feature vectors with spatial relationship semantic sharpening. These feature vectors have stronger semantic meaning of spatial relationships and can further improve the accuracy of the target image output by the global retrieval.
[0053] In one embodiment, such as Figure 2 As shown, a multimodal image retrieval method based on multi-granularity keyword parsing is provided. This method includes the following steps:
[0054] Step 201: Perform multi-granularity analysis on the query text input by the user through a large model, identify elements of different granularities in the query text, and decompose each element into multi-granularity keywords.
[0055] Step 202: Multi-granular keywords are used as semantic labels and input into the CLIP model for feature vector extraction. By comparing the cosine similarity between the feature vectors of the semantic labels and the feature vectors of each image stored in the database, several highly relevant images corresponding to each semantic label are selected.
[0056] Step 203: Select all highly relevant images as candidate images, input the candidate images and the complete query text into the LLM model for comprehensive semantic matching and reordering, and output the most relevant image of the query text. Further calculate and determine whether the similarity between the feature vector of the most relevant image and the feature vector of the query text is lower than a preset threshold. If it is lower, call the CLIP model again to perform a global search on all images in the database and return the image with the highest similarity to the feature vector of the query text as the target image; otherwise, directly use the most relevant image output by the LLM model as the target image.
[0057] In one embodiment, the method further includes: calling a translation API to translate the user-input query text into a language that the large model is adapted to, and inputting the translated query text into the large model for multi-granularity analysis; calling the CLIP model to extract the feature vector of each image in the database, and storing the feature vector of each image in the database.
[0058] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0059] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application.
Claims
1. A multimodal image retrieval system based on multi-granularity keyword parsing, characterized in that, The system consists of a language parsing module, a multimodal retrieval module, and a semantic reordering module connected in sequence. The language parsing module is used to perform multi-granularity analysis on the query text input by the user through a large model, identify elements of different granularities in the query text, and decompose each element into multi-granularity keywords. The multimodal retrieval module is used to take the multi-granular keywords output by the language parsing module as semantic tags and input them into the CLIP model for feature vector extraction. By comparing the cosine similarity between the feature vector of the semantic tag and the feature vector of each image stored in the database, several highly relevant images corresponding to each semantic tag are selected. The semantic reordering module is used to take all highly relevant images output by the multimodal retrieval module as candidate images, input the candidate images and the complete query text into the LLM model for comprehensive semantic matching and reordering, and output the most relevant image of the query text. It further calculates and determines whether the similarity between the feature vector of the most relevant image and the feature vector of the query text is lower than a preset threshold. If it is lower, the CLIP model is called again to perform a global search on all images in the database, and the image with the highest similarity to the feature vector of the query text is returned as the target image; otherwise, the most relevant image output by the LLM model is directly used as the target image.
2. The multimodal image retrieval system based on multi-granularity keyword parsing according to claim 1, characterized in that, Before performing multi-granularity analysis on the user-input query text through the large model, the language parsing module is also used to: call the translation API to translate the user-input query text into the language that the large model is adapted to process, and input the translated query text into the large model for multi-granularity analysis.
3. The multimodal image retrieval system based on multi-granularity keyword parsing according to claim 1 or 2, characterized in that, In the language parsing module, the large model identifies elements of different granularities, including objects, attributes, and spatial relationships in the query text.
4. The multimodal image retrieval system based on multi-granularity keyword parsing according to claim 1, characterized in that, Before comparing the cosine similarity between the feature vectors of semantic tags and the feature vectors of each image stored in the database, the multimodal retrieval module is also used to: call the CLIP model to extract the feature vectors of each image in the database, and store the feature vectors of each image in the database.
5. The multimodal image retrieval system based on multi-granularity keyword parsing according to claim 4, characterized in that, The multimodal retrieval module is also used to: create and start an independent thread for each semantic label and each image, and in each thread, call the CLIP model to extract the corresponding feature vector.
6. The multimodal image retrieval system based on multi-granularity keyword parsing according to claim 1, characterized in that, In the semantic reordering module, calculating and determining whether the similarity between the feature vector of the most relevant image and the feature vector of the query text is lower than a preset threshold includes: The CLIP model is invoked to extract the feature vector of the query text, and it is determined whether the cosine similarity between the feature vector of the query text and the feature vector of the most relevant image stored in the database is lower than a preset threshold.
7. The multimodal image retrieval system based on multi-granularity keyword parsing according to claim 6, characterized in that, In the semantic reordering module, the CLIP model is invoked again to perform a global search on all images in the database, and the image with the highest similarity to the feature vector of the query text is returned as the target image, including: The query text is input into the LLM model again, and the LLM model is used to extract the text describing spatial relationships in the query text and extract the corresponding feature vectors. The feature vectors of the text describing spatial relationships are superimposed with the feature vectors of the query text to generate a text feature vector with spatial relationship semantic sharpening. The CLIP model is then called again for global retrieval. The CLIP model is used to compare the cosine similarity between the text feature vector with spatial relationship semantic sharpening and the feature vectors of each image stored in the database. The image with the highest cosine similarity to the text feature vector with spatial relationship semantic sharpening is selected as the target image.
8. A multimodal image retrieval method based on multi-granularity keyword parsing, characterized in that, The method includes: By using a large model to perform multi-granularity analysis on the query text input by users, elements of different granularities in the query text are identified and each element is decomposed into multi-granularity keywords. Multi-granular keywords are used as semantic labels and input into the CLIP model for feature vector extraction. By comparing the cosine similarity between the feature vectors of the semantic labels and the feature vectors of each image stored in the database, several highly relevant images corresponding to each semantic label are selected. All highly relevant images are selected as candidate images. The candidate images are then input into the LLM model along with the complete query text for comprehensive semantic matching and reordering. The most relevant image for the query text is output. The similarity between the feature vector of the most relevant image and the feature vector of the query text is further calculated and determined to be lower than a preset threshold. If it is lower, the CLIP model is invoked again to perform a global search on all images in the database, and the image with the highest similarity to the feature vector of the query text is returned as the target image. Otherwise, the most relevant image output by the LLM model is directly used as the target image.
9. The multimodal image retrieval method based on multi-granularity keyword parsing according to claim 8, characterized in that, The method further includes: The translation API is called to translate the user-input query text into the language that the large model is adapted to, and the translated query text is then input into the large model for multi-granularity analysis. The CLIP model is called to extract the feature vector of each image in the database, and the feature vector of each image is stored in the database.