A cross-modal image retrieval method based on a multi-modal large language model intelligent agent

Abstract

The present application relates to the technical field of computer vision, and especially discloses a cross-modal image retrieval method based on a multi-modal large language model intelligent agent, which comprises the following steps: S1, initial candidate retrieval is performed by using a CLIP model; S2, attribute structural decomposition and difficulty determination are performed on a query text; S3, identity clustering is performed on candidate images by actively calling a DBSCAN clustering by the intelligent agent, and attribute reasoning verification is performed on multi-view images after clustering by calling a multi-modal large language model; S4, when there are competitive candidates, ambiguity is resolved by using part fine-grained scoring or direct visual inspection; and S5, a structured retrieval result is output. By introducing the DBSCAN unsupervised clustering technology, the present application compresses the redundant candidate space, enhances the reasoning confidence of the VLM through multi-view feature enhancement, and accurately distinguishes between multiple high-similarity candidates through part-level fine-grained scoring and active visual inspection.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a cross-modal image retrieval method based on a multimodal large language model intelligent agent. Background Technology

[0002] Cross-modal image retrieval aims to retrieve semantically matching images from large-scale image databases based on natural language descriptions, and has significant application value in fields such as security monitoring, criminal investigation assistance, and e-commerce. In recent years, methods based on cross-modal embedding models such as Contrastive Language Image Pre-training (CLIP) have made significant progress, achieving efficient cross-modal matching by mapping images and text to a shared semantic space and calculating similarity. However, existing methods still face the following technical challenges: single-round retrieval struggles to capture fine-grained attributes (such as accessories and shoe styles), viewpoint redundancy in multi-camera environments leads to duplicate candidate results, low-resolution image feature extraction is difficult, and retrieval decisions lack interpretability.

[0003] To address the aforementioned issues, existing research has attempted to enhance image retrieval by incorporating Large Language Models (LLM). For example, Chinese Invention Patent Publication No. CN120196777B discloses a fine-grained clothing image retrieval method and apparatus infused with common-sense knowledge from a Large Language Model. This method utilizes LLM to generate common-sense descriptions of attributes to enrich attribute semantics, aligns image patches with attribute descriptions through an attribute-guided cross-modal attention mechanism, and designs an imputation mechanism for modal missing elements. This approach has achieved good results in clothing image retrieval. However, it still has the following areas for improvement: (1) Its LLM is only used as an offline knowledge generator and lacks the autonomous decision-making ability to dynamically schedule different tools according to the retrieval confidence. The retrieval process is fixed and cannot adapt to queries of different difficulty. (2) This method only processes a single image and does not consider the viewpoint redundancy problem in the multi-camera environment. When multiple viewpoint images of the same target appear in the candidate list at the same time, it cannot perform identity positioning and joint reasoning. (3) It does not involve the enhancement processing of low-resolution images, and the recognition accuracy is limited in low-quality image scenarios such as surveillance. (4) The retrieval results are only similarity ranking and do not provide fine-grained attribute matching evidence, which lacks interpretability. Summary of the Invention

[0004] The purpose of this invention is to provide a cross-modal image retrieval method based on a multimodal large language model intelligent agent. This method can perform structured attribute decomposition through query statements and compare fine-grained attributes such as top and bottom garments one by one using a multimodal large model, thereby improving retrieval precision. Through multi-view clustering verification, it reduces result redundancy and improves coverage. While outputting retrieval results, it can provide structured evidence including details of matching / non-matching attributes and reasoning explanations. Furthermore, through fine-grained scoring at the part level and active visual inspection, it can accurately distinguish between multiple highly similar candidates.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a cross-modal image retrieval method based on a multimodal large language model intelligent agent, comprising the following steps: S1. Initial candidate image retrieval: Based on the input natural language query text, the CLIP model is used to calculate the similarity between the query text and the image, and a list of candidate images with the top K similarity ranking is extracted. S2. Query Attribute Decomposition and Difficulty Determination: Semantic parsing of natural language query text, extraction of structured attribute representations and generation of a discriminative attribute priority list, and calculation of query difficulty index. S3. Multi-view clustering verification: Perform DBSCAN identity clustering on candidate images based on visual features, grouping images of the same identity from different perspectives into one category. After selective super-resolution image enhancement for each cluster, call a fine-tuned multimodal large language model to perform multi-view attribute reasoning verification, and output the semantic verification conclusion of each identity cluster. When all clusters do not match, perform iterative query expansion based on the feedback of the non-matching attributes. S4. Fine-grained evaluation: When multiple competitive clusters are returned, the List-wise scoring mechanism is used to call the scoring model to score each competitive candidate by body part and calculate the comprehensive score. When the output of S3 determines that only some attributes match, the agent's visual perception ability is called to perform subjective visual confirmation of the candidate image. S5. Final Decision Output: Based on the verification conclusion of S3 and combined with the ambiguity resolution result of S4, output the optimal candidate image.

[0006] Preferably, the candidate image list in S1 includes image paths, pedestrian identifications, and similarity scores, and a similarity threshold is determined by a grid search method. When the similarity is lower than the threshold, a fine-tuning process is forcibly triggered.

[0007] Preferably, the structured attributes in S2 include gender, body type, description of upper garment, description of lower garment, description of footwear, description of hairstyle, and a list of accessories. The discriminative attribute priority list is assigned decreasing weights in the order of accessories, footwear and hair color, and clothing.

[0008] Preferably, the formula for calculating the query difficulty index in S2 is:

[0009] in: [ ] indicates an indicator function, for example, when the condition is met. hour, The value is 1 if it is set to 1, and 0 otherwise. CLIP score for Top-1 candidate; This indicates the number of valid attributes extracted by S2; This indicates the number of highly discriminative attributes; if there are no highly discriminative attributes, it is considered difficult. =0.46; =3; = = =1; according to The query is divided into three levels: easy, medium, and hard, corresponding to three strategies: skipping cluster validation, performing cluster validation but skipping fine-grained scoring of parts, and performing full cluster validation and fine-grained scoring of parts, respectively.

[0010] Preferably, in S3, selective super-resolution image enhancement is performed based on the image pixel area: when the pixel area is less than a first threshold or greater than or equal to a second threshold, super-resolution enhancement is performed; otherwise, enhancement is skipped.

[0011] Preferably, in S3, the multi-perspective attribute reasoning verification uses low-temperature parameters to generate structured JSON judgment conclusions. Each cluster independently calls the reasoning once, and outputs verification results including reasoning conclusions, confidence levels, details of matching and non-matching attributes, and reasoning explanations.

[0012] Preferably, the iterative query expansion mechanism in S3 includes: when all candidate clusters are not matched, summarizing the mismatched attributes returned by each cluster, identifying the set of mismatched attributes that frequently appear across clusters, concatenating the discriminative attributes in the attribute set to the beginning of the original query text to construct an enhanced query, re-executing the initial candidate retrieval and taking the union with the original candidates, and then performing cluster verification again.

[0013] Preferably, in S4, the List-wise scoring mechanism calls the scoring model for each part of the garment, such as the top, bottom, footwear, hairstyle, and accessories. The scoring model outputs the logarithmic probability of the yes or no token and normalizes it with Softmax to obtain the matching probability of each part. Only the average value of the parts actually included in the description in the query is taken as the comprehensive matching degree.

[0014] Preferably, the direct visual inspection in S4 includes: inserting up to 4 candidate images into the agent dialogue message stream in URI format, carrying a focus question to guide the agent to perform visual reasoning and focus on specific attributes, and calling the function up to 2 times per query.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention decomposes the query statement into structured attributes and uses a multimodal large model to compare fine-grained attributes such as tops, bottoms, footwear, and accessories one by one. This enables the model to capture key discriminative information such as clothing style and accessory details, thereby improving the precision of the retrieval.

[0016] 2. This invention introduces DBSCAN unsupervised clustering technology to group images of the same identity from different perspectives. This not only effectively compresses the redundant candidate space but also enhances the inference confidence of VLM through multi-view feature complementarity, enabling the system to maintain stable recognition capabilities under complex perspective changes.

[0017] 3. This invention, through the natural language generation capabilities of intelligent agents, can provide structured evidence, including details of matching / non-matching attributes and reasoning explanations, while outputting search results.

[0018] 4. This invention enables precise differentiation among multiple highly similar candidates through fine-grained location-level scoring and active visual inspection. By calculating the average location score and combining it with the agent's subjective perception, the system effectively eliminates retrieval ambiguities, ensuring that the final Rank-1 output has extremely high accuracy. Attached Figure Description

[0019] Figure 1 This is a flowchart of the retrieval method in this invention; Figure 2 This is a schematic diagram illustrating the different fine-grained ranking processes performed under varying query difficulty in this invention; Figure 3 This is the curve showing the decrease in fine-tuning loss in this invention; Figure 4 The curve showing the change in the accuracy of the output tokens in the model fine-tuning of this invention; Figure 5 This is a schematic diagram of cluster-guided multimodal large model inference in this invention. Detailed Implementation

[0020] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0021] Please refer to Figures 1-5 , a cross-modal image retrieval method based on a multi-modal large language model agent, comprising the following steps: S1. Initial candidate image retrieval.

[0022] According to the input natural language query text, locate the corresponding record in the CLIP similarity file, and extract the candidate image list with the top K (K = 20 in this embodiment) similarity rankings. Each candidate record includes: image path img_path, pedestrian identity identifier pid, and similarity score score.

[0023] Among them, S1 specifically includes: S11. Index positioning: First, accurately search with the query index; if not provided, after normalizing the query text, perform exact string matching; S12. Candidate cropping: Take the first K records from the complete similarity list to construct a candidate array with a ranking field; S13. Score threshold search: Determine the similarity threshold through grid search. When score < threshold (score threshold), the agent determines that the retrieval confidence is low, and the subsequent refined ranking process will be forced to trigger. The pre-computed formula for CLIP similarity is as follows:

[0024] Among them, is the embedding vector of the query text by the text encoder, is the embedding vector of the th image in the image library by the image encoder, is the cosine similarity score.

[0025] S2. Query attribute decomposition and difficulty determination.

[0026] Perform semantic parsing on the input natural language query, extract the structured attribute representation, and generate a discriminative attribute priority list to guide the focus of attention in the subsequent refined ranking steps; model and quantify the difficulty level of the input query to guide the dynamic refined ranking process.

[0027] Among them, S2 specifically includes: S21. Attribute Extraction: Retrieve the corresponding records from the pre-computed attribute decomposition file and extract the following attribute fields: gender: sex; build: body shape; upper: Description of the upper garment (color, style, material); lower: Description of lower garment; shoes: description of footwear; hair: Hairstyle description; accessories: A list of accessories (each accessory has an independent description field).

[0028] S22. Discriminative Ranking: Sort the attributes according to the following priority weights to generate a discriminative attribute list: Accessories (backpacks, hats, glasses, etc.) have the lowest probability of appearing in the crowd, so they are given the highest discrimination weight of 3; footwear and hair color are next, with a weight of 2; clothing has the lowest weight of 1 because of its wide color distribution. The first item of the sorted list will be passed as the focus parameter to step S3, guiding VLM to first check the most discriminative attribute.

[0029] S23. Part Query Construction: Organize the attribute descriptions into a part_queries dictionary with the structure {part name: natural language description of the part}, for use in the fine-grained scoring of step S41.

[0030] S24. Query difficulty determination and dynamic ranking: Please see Figure 2 This diagram illustrates different fine-tuning processes performed under varying query difficulty; a query difficulty index is defined. Three dimensions were considered:

[0031] in: [ ] indicates an indicator function, for example, when the condition is met. hour, The value is 1 if it is set to 1, and 0 otherwise. The CLIP score is the score of the Top-1 candidate. A score less than the threshold indicates that the CLIP model is not confident enough in its judgment. This represents the number of valid attributes extracted from S2; fewer attributes indicate a vague description. The number of highly discriminative attributes determines the difficulty; if there are no highly discriminative attributes, it is considered difficult. =0.46; =3; = = =1.

[0032] according to The value of adapts to adjust the depth of subsequent fine-tuning processes. Please refer to Table 1 for the subsequent fine-tuning process table.

[0033] Table 1 Subsequent Refinement Process Table ; S3, Multi-view clustering verification.

[0034] Using the query text as the sole input, the tool automatically completes four sub-processes: candidate loading, identity clustering, selective image enhancement, and multi-view VLM reasoning, and outputs the semantic verification conclusion for each competing identity.

[0035] Specifically, S3 includes: S31. Construction of training samples.

[0036] Based on the output of the initial retrieval model, a sample set is constructed for training the multimodal validation model, specifically including: S311, Generation of candidate set.

[0037] For any query text q, the similarity between the text and the images in the image library is calculated using a pre-trained ranking model (RDE model), resulting in a sequence of candidate images ranked by similarity. The top N candidates are then selected as the candidate set.

[0038] Where N=10.

[0039] S312, Identity Clustering Construction.

[0040] The images in the candidate set are clustered based on visual features, and images from different perspectives belonging to the same pedestrian are grouped into the same cluster, resulting in a candidate cluster set:

[0041] Each cluster A collection of images representing the same person, where the images contain identity information from different perspectives.

[0042] S313, Positive and Negative Sample Generation.

[0043] Based on the correspondence between query text and candidate clusters regarding real identities, supervised learning samples are constructed, including: positive samples (query text q and corresponding real identity cluster G+) and negative samples (query text q and non-corresponding identity cluster G-). Negative samples are preferably selected from candidate clusters with high similarity ranking but inconsistent identity labels to form difficult negative samples. Each sample contains a maximum of three images.

[0044] The training samples are from multiple public datasets, including CUHK-PEDES, RSTPReid, and ICFG-PEDES datasets. A unified training set is constructed using these datasets, with a total sample size of approximately 140,000 samples, to enhance the generalization ability of the model.

[0045] S32, Model fine-tuning training.

[0046] This invention employs a multimodal large model fine-tuning framework to optimize the parameters of a pre-trained model, enabling the model to possess text-driven cross-modal discrimination capabilities. Please refer to [link to relevant documentation]. Figure 3 This invention fine-tunes the loss reduction curve. Figure 4 The curve showing the change in the accuracy of the output tokens in the model of this invention is shown.

[0047] A multimodal large model with vision-language joint modeling capabilities is selected as the base model, preferably a Qwen series vision-language model, including but not limited to the Qwen-VL or Qwen3-VL-Instruct series models. The model includes at least: a Vision Encoder layer for extracting visual features from the input image; and a Large Language Model (LLM) layer for processing textual information and performing cross-modal reasoning.

[0048] Different parameter update strategies are adopted for different components of the multimodal model: All parameters of the visual encoding layer are updated to enable the model to meet the requirements of low-resolution, multi-view, and fine-grained attribute recognition across modal scenes. The formula is as follows:

[0049] in, Indicates the parameters of the visual coding layer; The learning rate is the step size for updating the model. This represents the gradient of the loss function with respect to the parameters.

[0050] The language model layer is fine-tuned efficiently using the low-rank adaptation (LoRA) method. This involves freezing most of the original parameters and introducing a low-rank trainable matrix, as shown in the following formula:

[0051] in, The original weight matrix, Let A be a low-rank matrix and r be its low-rank dimension. By training only A and B, task adaptation of the language model's reasoning capabilities is achieved.

[0052] During training, a distributed training strategy and mixed-precision computation are employed to improve training efficiency, and the following training control parameters are set to ensure stable model convergence: Number of training rounds: 2 rounds; Learning rate: ; Batch size: 64; Low-rank adaptive rank: 8; Low-rank adaptive scaling factor: 16; Maximum input length: 8192; Preheating ratio: 0.05; Please refer to Table 2 for the accuracy of the Qwen series multimodal models on three commonly used datasets. This invention uses the Qwen3-VL-8B-Instruct model as the base model. As shown in Table 2, this invention (Qwen3-VL-8B-Instruct) has the highest average accuracy on the three commonly used datasets.

[0053] Table 2. Accuracy of Qwen series multimodal models on three commonly used datasets. ; S33, Identity Clustering.

[0054] The top 20 candidate images are extracted based on the similarity ranking pre-computed by CLIP, and the candidates are grouped by pedestrian identity using DBSCAN clustering labels.

[0055] Offline clustering stage: CLIP visual features are extracted from the Top-50 candidate images for each query, and unsupervised clustering is performed using the DBSCAN algorithm (based on Euclidean distance in the feature space). Each image is assigned a cluster ID; images of the same pedestrian from different perspectives are clustered into the same cluster, and different pedestrians belong to different clusters. The clustering results are persisted as a JSON file for online querying. Online clustering stage: Given a set of candidate images The cluster ID corresponding to each image is found in the pre-computed index by image path, and images with the same cluster ID are merged into one identity cluster:

[0056] Each cluster selects the image with the highest similarity score as its representative image. All clusters are then sorted in descending order of their representative image scores, with the top-ranked images being selected. (In this embodiment, 5 clusters are entered into subsequent verification, effectively compressing 20 redundant candidates into several independent competing identities.)

[0057] S34. Selective super-resolution image enhancement based on stable diffusion.

[0058] For each cluster, a maximum of 3 images are selected. Whether to perform super-resolution enhancement is determined based on the pixel area of ​​the images, and the calculation formula is as follows:

[0059] in, =6000, =30000 (pixels) 2 All of these are the optimal thresholds obtained from grid search experiments.

[0060] The reason for skipping the middle interval is that when super-resolution models enlarge medium-resolution images, they tend to generate details that conform to their training distribution (such as smooth clothing textures and blurred accessory outlines). These generated contents are not real information of the image, which will cause VLM to make illusory judgments and reduce the verification accuracy.

[0061] S35, Multi-perspective VLM attribute reasoning verification.

[0062] For each identity cluster, its enhanced multi-view image is combined with the query text to construct a multimodal prompt. The inferential VLM (Qwen3-VL-8B-Instruct+SFT) is called to perform attribute-level comparison and inference. The inferential VLM (Qwen3-VL-8B-Instruct+SFT) is obtained by combining supervised fine-tuning of SFT on the basic model to obtain the final inferential VLM used.

[0063] The template structure is as follows: [System prompt] 'You are a person re-identification expert. ', 'You willbe shown one or more images of the SAME person from different viewpoints. ', 'Your task is to determine whether this person matches the given textdescription.' [User input] 'Text description of the target person:{query}"', '{focus_line}', 'The following {n_imgs} image(s) show the same person from differentviewpoints.', 'Carefully examine ALL attributes mentioned in the description.', 'For each attribute mentioned, state whether it matches or not.', 'Thengive your final verdict.' The inference-based VLM generates structured JSON judgments at a low temperature (temperature=0.1), with inference called independently once for each cluster. Finally, S3 outputs a list of all validated clustering results, with each record containing the inference conclusion, confidence level, details of matching and non-matching attributes, and an inference description.

[0064] S36. Iterative query expansion mechanism based on feedback of mismatched attributes.

[0065] When step S3 fails to find a match for all candidate clusters, the existing process cannot expand the search scope, resulting in recall failure. This is a systemic defect of CLIP when the initial search accuracy is insufficient.

[0066] In such cases, this invention first summarizes the list of mismatched attributes returned by all clustering VLM inferences, counts frequently occurring mismatched attributes across clusters, and identifies the set of "attributes where none of the candidates match". For example, if none of the candidates have "red backpack" but the query requires carrying a red backpack, then "red backpack" will be included. Next, execute the query attribute enhancement rewrite, which will... The discriminative attributes in the query text are appended to the beginning of the original query text to construct an enhanced query. The specific calculation formula is as follows:

[0067] in, The original text query is input to the model.

[0068] Enhanced query Resubmit to the CLIP retrieval module of S1 to obtain new Top-10 candidates, take the union with the original candidates to remove duplicates, and then execute the S3 clustering verification process again.

[0069] S4. Fine-grained evaluation: When multiple competitive clusters are returned, the List-wise scoring mechanism is used to call the scoring model to score each competitive candidate by body part and calculate the comprehensive score. When the output of S3 determines that only some attributes match, the agent's visual perception ability is called to perform subjective visual confirmation of the candidate image. S41, List-wise fine-grained scoring of body parts.

[0070] When S3 returns multiple competing clusters, the dedicated scoring VLM (Qwen3-VL-Reranker) is called to score the representative images of each competing candidate by body part, and a comprehensive score is calculated to resolve ambiguity.

[0071] Specifically, S41 includes: S411, Part-Level Scoring: For each candidate image, for each part (top, bottom, footwear, hairstyle, and accessories), a dedicated scoring VLM is invoked using part-specific cue words (to mask interference from other parts), and the matching probability of that part is output. The specific calculation formula is as follows:

[0072] in, , The log probabilities (logprobs) of the yes or no tokens output by the scoring VLM are normalized to probability values ​​using Softmax. This method is more stable than directly reading the generated text and is unaffected by changes in the model's output format.

[0073] By employing a list-wise scoring mechanism, the objective function at the ranking list level is directly optimized through overall modeling of the candidate set. Compared to point-wise or pair-wise methods, this approach can more accurately characterize the global relative relationships and importance distribution among samples, thereby reducing the problem of inconsistency between local optima and ranking. At the same time, this method is consistent with the requirement for overall ranking of the candidate list in actual reranking scenarios, which helps to fully utilize the semantic comparison information across samples and improve the discriminative power and stability of the ranking.

[0074] S412. Overall Score: The average score of each part is taken as the overall matching degree. The specific calculation formula is as follows:

[0075] Where P is the set of parts actually included in the query description (only parts with valid descriptions are scored to avoid introducing noise from irrelevant parts). The candidate with the highest overall score is the final recommendation result.

[0076] S42, Direct visual inspection.

[0077] When the S3 output determines that only some attributes match, the agent directly invokes its own visual perception ability to inject the best candidate image into its own context for subjective visual confirmation.

[0078] Specifically, S42 includes: S421, Image Injection: Insert up to 4 candidate images into the agent's dialogue message stream in file: / / URI format to trigger VLM's multimodal perception; S422, Focused Question Guidance: Can carry focused questions (such as "Does this person carry a red backpack?") to guide the intelligent agent's visual reasoning to focus on specific attributes; S423. Call limit: Each query can be called a maximum of 2 times, and a maximum of 4 images can be passed in each time to prevent context overload from causing a decrease in inference quality.

[0079] S5, Final Decision Output.

[0080] Based on the combined results of verification in S3 and ambiguity resolution in S4, the representative image of the optimal candidate identity was selected. Please refer to [link / reference]. Figure 5 This diagram illustrates the clustering-guided multimodal large-scale model inference of this invention. The system mandates that `final_rank1_img` and `final_rank1_pid` must originate from the actual return values ​​of the tool call; the agent must not infer or fabricate them to ensure the traceability and credibility of the output. The final output is a structured final result: { "final_rank1_img": "<img_path> ", "final_rank1_pid": <pid>, "reasoning": "<reasoning explanation text>" } It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0081] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.< / pid>

Claims

1. A cross-modal image retrieval method based on a multi-modal large language model agent, characterized in that, Includes the following steps: S1. Initial candidate image retrieval: Based on the input natural language query text, the CLIP model is used to calculate the similarity between the query text and the image, and a list of candidate images with the top K similarity ranking is extracted. S2. Query Attribute Decomposition and Difficulty Determination: Semantic parsing of natural language query text, extraction of structured attribute representations and generation of a discriminative attribute priority list, and calculation of query difficulty index. S3. Multi-view clustering verification: Perform DBSCAN identity clustering on candidate images based on visual features, grouping images of the same identity from different perspectives into one category. After selective super-resolution image enhancement for each cluster, call a fine-tuned multimodal large language model to perform multi-view attribute reasoning verification, and output the semantic verification conclusion of each identity cluster. When all clusters do not match, perform iterative query expansion based on the feedback of the non-matching attributes. Selective super-resolution image enhancement determines whether to enhance based on the image pixel area: super-resolution enhancement is performed when the pixel area is less than the first threshold or greater than or equal to the second threshold; otherwise, enhancement is skipped. S4. Fine-grained evaluation: When multiple competitive clusters are returned, the List-wise scoring mechanism is used to call the scoring model to score each competitive candidate by body part and calculate the comprehensive score. When the output of S3 determines that only some attributes match, the agent's visual perception ability is called to perform subjective visual confirmation of the candidate image. S5. Final Decision Output: Based on the verification results of S3 and the ambiguity resolution results of S4, the optimal candidate image is output.

2. The cross-modal image retrieval method based on the multi-modal large language model intelligent agent according to claim 1, characterized in that: The candidate image list in S1 includes image paths, pedestrian identifications, and similarity scores. A similarity threshold is determined using a grid search method. When the similarity is below the threshold, a fine-tuning process is forcibly triggered.

3. The cross-modal image retrieval method based on a multimodal large language model intelligent agent according to claim 1, characterized in that: The structured attributes in S2 include gender, body type, description of upper garment, description of lower garment, description of footwear, description of hairstyle, and a list of accessories. The discriminative attribute priority list is assigned decreasing weights in the order of accessories, footwear and hair color, and clothing.

4. The cross-modal image retrieval method based on a multimodal large language model intelligent agent according to claim 1, characterized in that: The formula for calculating the query difficulty index in S2 is as follows: in: [ ] indicates an indicator function, which takes the value 1 when the condition within the brackets is met, and 0 otherwise; CLIP score for Top-1 candidate; This indicates the number of valid attributes extracted by S2; This indicates the number of highly discriminative attributes; if there are no highly discriminative attributes, it is considered difficult. =0.46; =3; = = =1; according to The query is divided into three levels: easy, medium, and hard, which correspond to three strategies: skipping cluster validation, performing cluster validation but skipping fine-grained scoring of parts, and performing full cluster validation and fine-grained scoring of parts, respectively.

5. The cross-modal image retrieval method based on a multimodal large language model intelligent agent according to claim 1, characterized in that: The multi-perspective attribute reasoning verification in S3 uses low-temperature parameters to generate structured JSON judgment conclusions. Each cluster independently calls the reasoning once, and outputs verification results including reasoning conclusions, confidence levels, details of matching and non-matching attributes, and reasoning explanations.

6. The cross-modal image retrieval method based on a multimodal large language model intelligent agent according to claim 1, characterized in that: The iterative query expansion mechanism in S3 includes: when all candidate clusters are not matched, summarizing the mismatch attributes returned by each cluster, identifying the set of mismatch attributes that frequently appear across clusters, concatenating the discriminative attributes in the attribute set to the original query text to construct an enhanced query, re-executing the initial candidate image retrieval and taking the union with the original candidate image list, and then performing cluster verification again.

7. The cross-modal image retrieval method based on a multimodal large language model intelligent agent according to claim 1, characterized in that: In S4, the List-wise scoring mechanism calls the scoring model for each part of the garment, including the top, bottom, footwear, hairstyle, and accessories. The scoring model outputs the logarithmic probability of the yes or no token and normalizes it using Softmax to obtain the matching probability of each part. Only the average value of the parts actually included in the description in the query is taken as the comprehensive matching degree.

8. The cross-modal image retrieval method based on a multimodal large language model intelligent agent according to claim 1, characterized in that: The subjective visual confirmation in S4 includes: inserting up to 4 candidate images into the agent dialogue message stream in URI format, carrying a focus question to guide the agent to perform visual reasoning and focus on specific attributes, and calling the function up to 2 times per query.

Images