A cross-modal image retrieval method based on a multi-modal large language model intelligent agent

Abstract

This invention relates to the field of computer vision technology, and more particularly to a cross-modal image retrieval method based on a multimodal large language model (VLM) agent. The method includes: S1, initial candidate retrieval using the CLIP model; S2, attribute-structured decomposition and difficulty determination of the query text; S3, identity clustering of candidate images by actively calling DBSCAN clustering, and attribute inference verification of the clustered multi-view images by calling the multimodal large language model; S4, resolving ambiguity by using fine-grained location scoring or direct visual inspection when competing candidates exist; and S5, outputting structured retrieval results. This invention compresses redundant candidate space by introducing DBSCAN unsupervised clustering technology, enhances the inference confidence of the VLM through multi-view features, and accurately distinguishes between multiple highly similar candidates through fine-grained location scoring and active visual inspection.

Claims

1. A cross-modal image retrieval method based on a multimodal large language model intelligent agent, characterized in that, Includes the following steps: S1. Initial candidate image retrieval: Based on the input natural language query text, the CLIP model is used to calculate the similarity between the query text and the image, and a list of candidate images with the top K similarity ranking is extracted. S2. Query Attribute Decomposition and Difficulty Determination: Semantic parsing of natural language query text, extraction of structured attribute representations and generation of a discriminative attribute priority list, and calculation of query difficulty index. S3. Multi-view clustering verification: Perform DBSCAN identity clustering on candidate images based on visual features, grouping images of the same identity from different perspectives into one category. After selective super-resolution image enhancement for each cluster, call a fine-tuned multimodal large language model to perform multi-view attribute reasoning verification, and output the semantic verification conclusion of each identity cluster. When all clusters do not match, perform iterative query expansion based on the feedback of the non-matching attributes. S4. Fine-grained evaluation: When multiple competitive clusters are returned, the List-wise scoring mechanism is used to call the scoring model to score each competitive candidate by body part and calculate the comprehensive score. When the output of S3 determines that only some attributes match, the agent's visual perception ability is called to perform subjective visual confirmation of the candidate image. S5. Final Decision Output: Based on the verification results of S3 and the ambiguity resolution results of S4, the optimal candidate image is output.

2. The cross-modal image retrieval method based on a multimodal large language model intelligent agent according to claim 1, characterized in that: The candidate image list in S1 includes image paths, pedestrian identifications, and similarity scores. A similarity threshold is determined using a grid search method. When the similarity is below the threshold, a fine-tuning process is forcibly triggered.

3. The cross-modal image retrieval method based on a multimodal large language model intelligent agent according to claim 1, characterized in that: The structured attributes in S2 include gender, body type, description of upper garment, description of lower garment, description of footwear, description of hairstyle, and a list of accessories. The discriminative attribute priority list is assigned decreasing weights in the order of accessories, footwear and hair color, and clothing.

4. The cross-modal image retrieval method based on a multimodal large language model intelligent agent according to claim 1, characterized in that: The formula for calculating the query difficulty index in S2 is as follows: ， in: [ ] indicates an indicator function, for example, when the condition is met. hour, The value is 1 if it is set to 1, and 0 otherwise. CLIP score for Top-1 candidate; This indicates the number of valid attributes extracted by S2; This indicates the number of highly discriminative attributes; if there are no highly discriminative attributes, it is considered difficult. =0.46; =3; = = =1; according to The query is divided into three levels: easy, medium, and hard, corresponding to three strategies: skipping cluster validation, performing cluster validation but skipping fine-grained scoring of parts, and performing full cluster validation and fine-grained scoring of parts, respectively.

5. The cross-modal image retrieval method based on a multimodal large language model intelligent agent according to claim 1, characterized in that: In S3, selective super-resolution image enhancement is performed based on the image pixel area: when the pixel area is less than the first threshold or greater than or equal to the second threshold, super-resolution enhancement is performed; otherwise, enhancement is skipped.

6. The cross-modal image retrieval method based on a multimodal large language model intelligent agent according to claim 1, characterized in that: The multi-perspective attribute reasoning verification in S3 uses low-temperature parameters to generate structured JSON judgment conclusions. Each cluster independently calls the reasoning once, and outputs verification results including reasoning conclusions, confidence levels, details of matching and non-matching attributes, and reasoning explanations.

7. The cross-modal image retrieval method based on a multimodal large language model intelligent agent according to claim 1, characterized in that: The iterative query expansion mechanism in S3 includes: when all candidate clusters are not matched, summarizing the non-matching attributes returned by each cluster, identifying the set of non-matching attributes that frequently appear across clusters, concatenating the discriminative attributes in the attribute set to the original query text to construct an enhanced query, re-executing the initial candidate retrieval and taking the union with the original candidates, and then performing cluster verification again.

8. The cross-modal image retrieval method based on a multimodal large language model intelligent agent according to claim 1, characterized in that: In S4, the List-wise scoring mechanism calls the scoring model for each part of the garment, including the top, bottom, footwear, hairstyle, and accessories. The scoring model outputs the logarithmic probability of the yes or no token and normalizes it using Softmax to obtain the matching probability of each part. Only the average value of the parts actually included in the description in the query is taken as the comprehensive matching degree.

9. The cross-modal image retrieval method based on a multimodal large language model intelligent agent according to claim 1, characterized in that: The direct visual inspection in S4 includes: inserting up to 4 candidate images into the agent dialogue message stream in URI format, carrying a focus question to guide the agent to perform visual reasoning and focus on specific attributes, and calling it up to 2 times for each query.

Images