A multi-round dynamic evolution intention-oriented conversational image retrieval method

Abstract

The application discloses a multi-round dynamic evolution intention-oriented conversational image retrieval method, which firstly constructs a user simulation framework based on a large model and a deep intention reasoning engine as a teacher model, reduces and purifies a multi-modal dialogue history into a de-contextualized text query through structured thinking chain reasoning, and synthesizes multi-round dialogue training data containing successful trajectories and failure trajectories through closed-loop interaction; then the multi-step reasoning ability of the teacher model is implicitly transferred to a lightweight student model through an RIDE embedding distillation framework, and combined with three difficult negative sample mining mechanisms and progressive ranking constraints, the two aspects of semantic level comparison alignment and trajectory level dynamic evolution are optimized; finally, the slow generation reasoning ability is distilled into the fast embedding retrieval model, so that the multi-round conversational image retrieval considering retrieval accuracy, long-range intention tracking ability and real-time response efficiency is realized through single forward propagation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a conversational image retrieval method oriented towards multi-turn dynamic evolution intentions. Background Technology

[0002] Visual data, as the most intuitive carrier for humans to perceive the world, record civilization, and transmit information, has experienced explosive growth in the digital and internet era. How to accurately and efficiently retrieve target images that match user intent from the vast sea of ​​visual data has always been a crucial problem and core challenge in the fields of computer vision and multimedia. Image retrieval is a core technology system developed to overcome this challenge. Its main purpose is to establish reliable feature measurement criteria and accurately match and recall strongly relevant visual images from massive databases based on user-input text or image query clues. For a long time, mainstream image retrieval research has largely followed a basic static snapshot paradigm: the system typically receives single, single-modal query clues, and uses this to complete coarse-grained alignment of macroscopic visual features. In recent years, with the rapid development of deep learning and artificial intelligence technologies, existing retrieval models have made significant progress in understanding visual images at the semantic feature level, greatly improving the efficiency of large-scale feature database construction and retrieval comparison.

[0003] In fact, the essence of human natural cognition and search practices has always been a dynamic process of constantly evolving intentions and continuously refined details. In the past, limited by the technical bottlenecks of retrieval systems, users were often forced to simplify their complex requests. With the explosion of multimodal artificial intelligence technology, this latent need for natural interaction has been fully stimulated. Users' search needs are no longer limited to single, static, coarse-grained matching, but rather exhibit an evolving expression from coarse to fine, from single to multidimensional, and from static to dynamic. For example, a user's initial search may only be to find a macroscopically similar category (such as searching for a cat); then, they will naturally request to pinpoint potentially easily confused sub-categories (such as specifying a particular Maine Coon breed); when they find a visual target that roughly matches their expectations, they will have a multimodal combination request to superimpose textual instructions for local fine-tuning (such as requesting to search for this cat, but with a darker coat); if the results are still not perfect, users often expect to receive multiple rounds of proactive guidance and correction based on the current results, just like in daily communication (such as proactively adding that it should also be wearing a red collar). In this evolutionary process, users are no longer satisfied with systems merely providing vaguely similar information. They expect systems to be able to keenly capture easily confused subtle differences, deeply understand overlapping and complex textual and image constraints, and empower users with sufficient control to guide and correct deviations. When these increasingly complex user demands, evolving from simple to complex, are compounded by the massive, long-tailed, and visually confusing data interference in the real world, traditional image retrieval systems expose their inherent limitations. Therefore, the core challenge of modern image retrieval systems has shifted from traditional macroscopic comparison of visual features to a refined response to multi-layered evolving intentions.

[0004] To accurately respond to the aforementioned multi-layered evolutionary intentions, image retrieval systems must overcome technical barriers ranging from basic visual representation and cross-modal semantic interaction to multi-round dynamic long-range tracking. This process faces unique and severe challenges.

[0005] First, achieving precise alignment of microscopic details in basic visual representations is extremely difficult. Whether it's a single visual query or a step-by-step image-text combination query, users require retrieval systems to not only identify the macroscopic subject but also keenly capture subtle differences in local visual attributes. Existing methods face two bottlenecks in this direction: Regarding the construction of difficult samples, mining-based methods essentially resample empirical data, lacking sample diversity, while sample synthesis methods based on generative networks or feature mixing rely excessively on local sample interactions, lacking global prior guidance on the macroscopic distribution of the feature manifold, and are prone to introducing noisy samples. Regarding metric space optimization, classic sample-pair-based losses face the dilemma of exploding multinomial computational complexity, while surrogate-based losses often sever the collaborative optimization of intra-class global structural representation and inter-class local detail differentiation. These limitations in representation learning prevent retrieval models from accurately decoupling and representing subtle local visual attributes, making it difficult to meet the stringent requirements of fine-grained matching.

[0006] Secondly, bridging the multimodal divide and achieving deep fusion of combined intents is extremely challenging. When retrieval intent evolves beyond unimodality and incorporates textual commands, the retrieval task is no longer a one-dimensional matching of visual features, but transforms into a complex multimodal combined semantic understanding—requiring not only the retrieval model to accurately understand discrete linguistic expressions, but also to deeply interact with and align them with continuous visual signals, accurately executing the local feature evolution of textual pointing to the reference image. Existing methods are caught in a dilemma between inference efficiency and fusion depth: on the one hand, inference-based methods, represented by multimodal thought chains, can achieve deep image-text semantic interaction, but their complex inference process leads to huge computational resource consumption, making it difficult to meet the efficiency requirements of practical applications; on the other hand, methods based on self-supervised training paradigms, while having advantages in efficiency, often fail to fully capture the complex semantic relationships between images and text due to their shallow fusion mechanisms, resulting in limited multimodal information fusion capabilities.

[0007] Furthermore, tracking dynamic intent in long-term dialogues involving multiple turns of interaction is difficult. Real-world dialogue retrieval processes exhibit dynamic evolution; users' search goals constantly shift and diverge based on multiple rounds of visual feedback, posing a significant challenge to the model's long-term memory and deep reasoning capabilities. On one hand, at the interaction paradigm level, existing dialogue models have consistently failed to perfectly align with real human interaction habits. Early conversational image retrieval efforts, limited by recurrent neural networks and other infrastructure, often only processed very shallow static contexts, not truly engaging in turn-by-turn dialogue, and task settings were often confined to vertical domains such as fashion and styling. While the introduction of large language models and the dominant system-based questioning mechanism broke domain limitations, it is essentially a passive interaction for users, who can only passively answer system questions without expressing their true needs, hindering true human-computer collaboration. On the other hand, at the level of long-term intent tracking, existing methods often lack specific design for long-term dynamic reasoning in their model architecture, making it difficult for models to continuously track and accurately respond to user intent. To address this, researchers have attempted to introduce techniques such as reinforcement learning and memory enhancement mechanisms to improve the dynamic tracking capabilities of models. However, these methods are prone to catastrophic forgetting and severe degradation of generalization ability when faced with complex dialogues in open worlds with long contexts.

[0008] Therefore, overcoming the three major technical challenges of "difficulty in fine-grained feature identification," "difficulty in deep fusion of multimodal information," and "difficulty in tracking long-range dialogue intent" in diverse open scenarios, in order to accurately analyze and respond to users' multi-layered and complex search intents and construct a universal visual retrieval paradigm that highly conforms to human interaction logic, has become a core technical problem that urgently needs to be solved in this field. The current field urgently needs a method that can simultaneously address the above three challenges: at the visual representation level, it needs to break through the bottleneck of coarse-grained representation and possess extremely fine-grained feature perception and identification capabilities; at the cross-modal interaction level, it needs to achieve deep semantic fusion and accurate alignment of image-text combination intents, while also considering reasoning efficiency; at the multi-turn dialogue level, it needs to empower users to actively guide the dialogue and continuously track and accurately respond to long-range evolving intents, thereby achieving high-precision and high-efficiency conversational image retrieval for multi-turn dynamically evolving intents. Summary of the Invention

[0009] (1) Technical problems to be solved This invention discloses a conversational image retrieval method for multi-turn dynamic evolution of intent, aiming to solve the problems of rigid interaction paradigm, passive user position, insufficient ability to track long-term intent in multi-turn interactions, and the contradiction between complex reasoning and real-time efficiency in existing conversational image retrieval methods.

[0010] (2) Technical solution This invention provides a conversational image retrieval method oriented towards multi-turn dynamically evolving intents. The method is based on reasoning-guided embedding distillation and includes the following steps: Step 1: Construct a multi-level classification system, build an open-world image library based on the multi-level classification system, and deploy a large-scale user simulation framework to generate multi-round conversational image retrieval training data. Step 2: Perform multi-round simulated retrieval interaction through the teacher model, generate decontextualized text queries based on the multimodal dialogue history to retrieve images from the library, and generate training data containing positive and difficult trajectories based on the retrieval results. Step 3: Build a RIDE embedding distillation framework with multiple functional modules as the student model, read the training trajectory output in Step 2, concatenate the multimodal input, and extract the global context embedding representing the retrieval intent. Step 4: Construct three types of difficult negative samples from three dimensions: historical interaction semantics, differentiated images of the same type in the image library, and candidate images from historical retrieval. Step 5: Construct a joint optimization objective using the three types of difficult negative samples and the global context embedding, iteratively optimize the student model, and complete the knowledge distillation from the teacher model to the student model; Step 6: Offline pre-encode the images in the image library to build vector indexes, online receive multi-turn user feedback to splice multimodal dialogue context, use the trained student model to extract context embeddings and retrieve vector indexes, iterate interactively until the target image is output.

[0011] Furthermore, the specific steps of step 1 are as follows: Step 101: Construct a multi-level classification system, dividing visual concepts into five core dimensions from an ontological perspective: nature, human living environment, man-made objects, human society, and events and processes. Step 102: Organize each of the core dimensions into a four-layer progressive structure of macro layer, meso layer, micro layer and specific category layer, and construct an open world image library based on the multi-level classification system; Step 103: Deploy a user simulation framework based on a large model. Based on the open-world image library, generate multi-turn conversational image retrieval training data through structured intent planning, human-like realistic behavior modeling, and closed-loop interaction with state awareness.

[0012] Further, step 103 specifically includes: Step 10301: Perform structured intent planning on user feedback behavior and classify intent templates into six core interaction patterns; Step 10302: Model the simulated user’s real-world behavior. Generate conversational and fragmented natural language feedback through layperson role setting and a progressive disclosure mechanism. Each round will reveal a maximum of a preset number of new modification details. Step 10303: Perform state-aware closed-loop interaction. Before generating each round of feedback, analyze the difference between the current candidate image and the target image, verify the consistency between the modification points to be fed back and the historically confirmed features, and select the intent template accordingly. Step 10304: Generate feedback text based on the selected intent template, and evaluate the quality of simulated user feedback from the dimensions of visual fidelity, language naturalness, and behavioral consistency.

[0013] Furthermore, step 2 is specifically as follows: Step 201: In the interactive round, the teacher model receives the multimodal dialogue history, which includes at least the Top-K candidate image set and text dialogue context of the previous round, and performs visual analysis and difference alignment on the multimodal dialogue history to form effective historical features; Step 202: Fuse the new visual constraints of the current round with the historical effective features to generate a decontextualized text query. Input the decontextualized plain text query into the visual-language retrieval device, retrieve the open-world image library for retrieval, and output the Top-K candidate image set for this round. If the target image is located in the Top-K candidate image set for this round, it is saved as a positive training trajectory. If it is not detected and the upper limit of the interaction round has not been reached, the user simulation framework is invoked to generate new feedback and iterate the interaction. If it is not detected and the upper limit of the round has been reached, the interaction trajectory is saved as a negative sample to construct a difficult trajectory.

[0014] Furthermore, step 3 includes the following steps: Step 301: Build a student model based on the RIDE embedding distillation framework, which includes a multimodal context embedding generation module, a dynamic hard negative sample mining module, a dynamic contrastive loss calculation module, and a progressive ranking constraint module. Step 302: Read the training trajectory output in Step 2, extract the text dialogue context from the trajectory as the accumulated text retrieval requirements, extract the Top-K candidate images of each round as the recall images of the previous round, and concatenate the two to form a multimodal input; feed the multimodal input into the student model, and extract the hidden layer state of the special marker words at the end of the sequence as the global context embedding.

[0015] Furthermore, step 4 includes the following steps: Step 401: Construct incomplete semantic negative samples based on historical interaction semantics, and use the previous round of multimodal context embedding and the previous round of decontextualized text query embedding as negative samples to participate in comparative learning, so as to force the model to encode the information increment of the current round; Step 402: Construct intra-class visual negative samples for differentiated images of the same type in the image library. Samples from the open-world image library that belong to the same semantic category as the target image but have differences in details are used to enhance fine-grained discrimination ability. Step 403: Construct interactive mining visual negative samples for historical retrieval candidate images. Use a multimodal large language model to divide the Top-K candidate image set output in each round of Step 2 into three-state classifications: positive samples, simple negative samples, and difficult negative samples. This is used to explicitly suppress historical erroneous retrieval results that violate user instructions.

[0016] Furthermore, the specific steps of step 5 are as follows: Step 501: Integrate the three sets of difficult negative samples generated in Step 4 through the dynamic difficult negative sample mining module to obtain dynamic difficult negative samples; Step 502: Calculate the dynamic contrastive loss based on global context embedding and dynamic hard negative samples, and calculate the progressive ranking constraint; Among them, dynamic contrast loss The calculation formula is as follows:

[0017] in, For the first Multimodal context embedding representation of the wheel; For target image Embedded representation obtained by image encoder; and These are the multimodal context embedding and decontextualized text query embedding from the previous round, respectively; To interactively mine the visual negative sample set; The set of visual negative samples within the class; Represents the current round's multimodal context embedding representation Embedded with target image Scaling cosine similarity; Represents the current round's multimodal context embedding representation Compared with the previous round of multimodal context embedding Scaling cosine similarity, Represents the current round's multimodal context embedding representation Compared with the previous round of decontextualized text query embedding Scaling cosine similarity; Represents the current round's multimodal context embedding representation With negative sample embedding The scaled cosine similarity, where, To mine visual negative sample sets from interactions With the in-class visual negative sample set Embed any negative sample taken from the set; Step 503: Construct a joint optimization objective by combining the dynamic contrast loss and the progressive ranking constraint, and iteratively optimize the student model parameters.

[0018] Furthermore, the progressive sorting constraint in step 502 Defined as: for consecutive rounds and The current round requires multimodal context embedding. Embedded with target image The similarity relative to the previous round of context embedding Embedded with target image The similarity should at least increase the preset margin. The formula is as follows:

[0019] in, For indicator functions, when The value is 1 if the condition is met; otherwise, the value is 0. Constrain margins for sorting; The cosine similarity between the current context embedding and the target image embedding; The progressive ranking constraint is used to force the similarity between the query feature and the target image to increase monotonically along the dialogue trajectory, based on the cosine similarity between the previous round of context embedding and the embedding of the same target image.

[0020] Furthermore, the joint optimization objective in step 503 is: ,in To balance the hyperparameters; by minimizing The multi-step reasoning ability of the teacher model is distilled into the student model.

[0021] Furthermore, step 6 specifically includes: Step 601: In the offline stage, the trained student model is used to pre-encode candidate images in the open-world image library to obtain the image embedding representation corresponding to each image, and a vector index is constructed based on the image embedding representation. Step 602: During the online interaction phase, receive the user's current round of text feedback and the candidate images returned in the previous round, and combine them with historical dialogue information to form the current round of multimodal dialogue context. Step 603: Input the multimodal dialogue context into the trained student model, and extract the final hidden layer state of the special marker word at the end of the sequence as the current round context embedding representation; Step 604: Based on the context embedding representation, perform similarity retrieval in the vector index to obtain and output the Top-K candidate images that match the current retrieval intent; Step 605: When the user continues to input new feedback information, the new feedback information and the current round of candidate images are incorporated into the multimodal dialogue context, and steps 603 to 604 are repeated until the target image that meets the user's retrieval needs is output.

[0022] The present invention has the following advantages over the prior art: The freedom of user-initiated interaction is significantly enhanced. This invention pioneers a dynamic dialogue simulation framework based on a "large model as the user," employing three main components: structured intent planning (23 intent templates, six interaction modes), human-like realistic behavior modeling (layperson role setting + progressive disclosure mechanism), and state-aware closed-loop interaction (four-step internal reflection and verification). This empowers users to actively guide the dialogue, fundamentally breaking through the limitations of existing system-based passive question-and-answer paradigms. Users are no longer passive respondents to system questions, but rather active guides in the retrieval process.

[0023] The long-term intent tracking capability is significantly enhanced. This invention introduces three dynamic hard negative sample mining mechanisms: incomplete semantic negative samples force the model to encode the current round's information increment and suppress outdated intent interference; intra-class visual negative samples sharpen fine-grained decision boundaries; and interactive visual negative sample mining employs an asymmetric weighting mechanism to avoid repeating historical retrieval errors—designed specifically for the temporal semantic evolution in dialogue scenarios, far superior to the coarse random negative sample or batch negative sample strategies in existing methods. Simultaneously, the progressive ranking constraint explicitly encodes the temporal dynamic characteristics of dialogue into the embedded learning objective for the first time, forcing a monotonically increasing retrieval quality across rounds, effectively preventing feature degradation and catastrophic forgetting in long-term interactions.

[0024] The contradiction between inference efficiency and retrieval accuracy is fundamentally resolved. The RIDE distillation framework proposed in this invention achieves the core innovation of "turning slow into fast": the single-round inference of the large teacher model, which requires tens of seconds (a two-stage paradigm of first generating a text query and then retrieving an image based on the query), can be accurately retrieved from a million-level image database in about 1.5 seconds after being distilled into a lightweight student embedding model by RIDE, requiring only a single forward propagation, with latency compression of more than 20 times. While significantly improving inference efficiency, retrieval accuracy does not decrease but rather increases—the 2B-parameter student model enhanced by RIDE significantly outperforms the 8B-parameter model without framework enhancement in the R@1 metric, demonstrating the significant advantage of the feature distillation strategy over the paradigm of simply relying on model size expansion.

[0025] A breakthrough in performance across parameter levels. The small-parameter student model, enhanced by distillation within the RIDE framework, significantly outperforms the large-parameter model without framework enhancement in overall retrieval performance. This provides a highly valuable lightweight path for the industrial deployment of real-time conversational image retrieval systems.

[0026] The training data is of high quality and scalability. The teacher model's deep intent reasoning engine uses a three-step structured thought chain reasoning to reduce the complexity of multimodal dialogue history into clear, independent retrieval needs. Combined with a dual-path closed-loop collection mechanism of successful and failed trajectories, it synthesizes a training corpus containing 10,000 high-quality multi-turn dialogues. Crucially, failed trajectories (dialogues that have reached the turn limit but have not been successfully retrieved) are included in the training set as challenging samples, significantly improving the model's robustness in real-world, complex scenarios.

[0027] The offline database construction and online inference processes are decoupled, resulting in strong engineering scalability. Image precoding and vector indexing are performed offline in a one-time process. During online interaction, only one forward propagation of the student model is needed to complete the vector retrieval within the current multi-round context. When the database size expands from millions to tens of millions, the main increases are in the overhead of offline feature storage, index construction, and vector retrieval, without introducing the linear latency of multimodal large-scale model round-by-round generative inference. This provides a clear engineering scalability path for low-latency interactive image retrieval systems.

[0028] This invention overcomes three core challenges in conversational image retrieval—rigid interaction paradigms, difficulty in intent tracking, and the efficiency-accuracy contradiction—through the collaborative innovation of a dynamic dialogue simulation framework based on a "large model as user," a deep intent reasoning engine, and a RIDE embedded distillation framework. It breaks through technical bottlenecks in retrieval accuracy, reasoning efficiency, and cross-generational adaptability, providing an end-to-end technical solution for the industrial deployment of interactive visual search systems. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the conversational image retrieval method of the present invention, which is designed for multi-round dynamic evolution.

[0030] Figure 2 This is a schematic diagram of the system structure according to an embodiment of the present invention.

[0031] Figure 3 This is a schematic diagram of three types of dynamic hard negative sample mining in an embodiment of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] like Figure 1-2 As shown, this invention discloses a conversational image retrieval method oriented towards multi-turn dynamic evolution intent. The method is based on reasoning-guided embedding distillation and includes the following steps: Step 1: Construct a data preparation module, including a multi-level classification system, build an open-world image library based on the multi-level classification system, and deploy a large-scale user simulation framework to generate multi-turn conversational image retrieval training data. Step 2: Construct a teacher model through a deep intent reasoning engine, conduct multi-round simulated retrieval interactions through the teacher model, perform visual difference alignment on the multimodal dialogue history containing historical candidate images and text dialogue context, integrate the newly added visual constraints to generate a decontextualized text query retrieval library, and generate positive training trajectories and difficult trajectories according to the retrieval hit rate and the upper limit of the interaction rounds. Step 3: Build a RIDE embedding distillation framework with multiple functional modules as the student model, read the training trajectory output from Step 2, concatenate the multimodal input, and extract the global context embedding representing the retrieval intent. Step 4: Construct three types of difficult negative samples from three dimensions: historical interaction semantics, differentiated images of the same type in the image library, and candidate images from historical retrieval. Step 5: Integrate the three types of difficult negative samples to obtain dynamic difficult negative samples. Combine global context embedding to calculate dynamic contrastive loss and set progressive ranking constraints. Use the two to form a joint optimization objective to iteratively optimize the student model. Implicitly transfer the multi-step reasoning ability of the teacher model to the feature space of the student model to complete knowledge distillation. Step 6: Offline pre-encode the images in the image library to build vector indexes, online receive multi-turn user feedback to splice multimodal dialogue context, use the trained student model to extract context embeddings and retrieve vector indexes, iterate interactively until the target image is output.

[0034] Specifically, step 1 consists of the following steps: Step 101: From an ontological perspective, visual concepts are divided into five core dimensions: nature, human living environment, man-made objects, human society, and events and processes. These five core dimensions are then organized into a four-layer progressive structure: macro level, meso level, micro level, and specific category level, forming a complete classification system covering 320 visual categories. Step 102: Based on the classification system, collect approximately 676,000 high-quality images from public image datasets and targeted web crawlers, and combine them with approximately 2.65 million general image data to construct an open-world candidate image library of approximately 3.3 million images; Step 103: Deploy a user simulation framework based on a large model. Select a large model with multimodal reasoning capabilities as the user simulator engine. Generate multi-turn conversational image retrieval training data through structured intent planning, human-like real behavior modeling, and closed-loop interaction with state awareness.

[0035] Step 103 specifically includes: Step 10301: Perform structured intent planning on user feedback behavior and classify intent templates into six core interaction patterns; the six core interaction patterns are progressive refinement, corrective feedback, attribute modification, comparison and exploration, intent transfer, and abstract semantic refinement. Step 10302: Model the simulated user’s real-world behavior. Generate conversational and fragmented natural language feedback through layperson role setting and a progressive disclosure mechanism. Each round will reveal a maximum of a preset number of new modification details. Step 10303: Perform state-aware closed-loop interaction. Before generating each round of feedback, analyze the difference between the current candidate image and the target image, verify the consistency between the modification points to be fed back and the historically confirmed features, and select the intent template accordingly. Step 10304: Generate feedback text based on the selected intent template, and evaluate the quality of simulated user feedback from the dimensions of visual fidelity, language naturalness, and behavioral consistency. Select high-quality, highly realistic multi-turn interactive feedback, which will ultimately be used to construct standardized, highly realistic multi-turn conversational image retrieval training data.

[0036] Specifically, step 2 consists of the following steps: Step 201: In each interaction round, the teacher model receives a multimodal dialogue history containing the Top-K candidate image set and text dialogue context from the previous round; based on the multi-level classification system, it performs visual analysis and difference alignment on the multimodal dialogue history to identify the semantic gap between candidate images and target images, and retains the valid historical features confirmed by the user; the teacher model's deep intent reasoning engine uses a three-step structured thought chain reasoning to reduce the dimensionality of the noisy multimodal dialogue history into clear independent retrieval requirements, and, in conjunction with a dual-path closed-loop collection mechanism of success and failure trajectories, synthesizes a training corpus containing 10,000 high-quality multi-turn dialogues.

[0037] Step 202: Integrate the historical valid features with the new visual constraints of the current round, filter out features that have been rejected by the user, and generate a decontextualized plain text query. The generated decontextualized plain text query is input into the visual-language retrieval device, which performs intensive retrieval in the 3.3 million-level open-world candidate image library constructed in step 102, and outputs the Top-K candidate image set for the current round; it is then determined whether the target image exists in the current Top-K candidate image set. If the target image is hit, the complete multi-round interaction tuple of this round is saved as a positive training trajectory; If the target image is not hit and the maximum number of interaction rounds is not reached, then step 103, the user simulator, is called to generate a new round of realistic human-like user feedback, the multimodal dialogue history is updated, and the next round of iterative retrieval begins. If the training continues to fail and the maximum number of interaction rounds is reached, the entire non-converged multi-round interaction trajectory is saved as a difficult failure training trajectory and incorporated into the subsequent training set as a difficult negative sample data source.

[0038] In one embodiment, the aforementioned closed-loop data synthesis mechanism can generate 10,000 multi-turn dialogue training corpora containing complete historical interaction trajectories.

[0039] Specifically, step 3 is as follows: Step 301: Construct a student model, which is a RIDE embedding distillation framework. The RIDE embedding distillation framework includes a multimodal context embedding generation module, a dynamic hard negative sample mining module, a dynamic contrastive loss calculation module, and a progressive ranking constraint module. In one implementation, the student model backbone network is Qwen3-VL-Embedding-2B, trained on eight NVIDIA RTX 4090 GPUs, and LoRA is used for efficient parameter fine-tuning.

[0040] Step 302: Read the training trajectory output in Step 2, extract the text dialogue context from the trajectory as the accumulated text retrieval requirements, extract the Top-K candidate images from each round as the recall images of the previous round, and concatenate the two to form a multimodal input; feed the multimodal input into the student model, and after feeding it into the student model, extract the final hidden layer state of the special marker words at the end of the sequence as the global context embedding. , ,in For the embedded vector dimension; In one implementation, the single-card batch size is 1, and each anchor point is associated with 12 hard negative samples.

[0041] Specifically, such as Figure 3 As shown, step 4 includes the following: Step 401: Construct incomplete semantic negative samples based on historical interaction semantics, embedding the previous round of multimodal context. Compared to the previous round of decontextualized text query embedding As negative samples, they participate in contrastive learning to force the model to encode the information increment of the current round and suppress interference from outdated intentions; Step 402: Construct intra-class visual negative samples for differentiated images of the same type in the image library. Samples belonging to the same semantic category as the target image but with differences in details are sampled from the open-world image library to enhance the model's fine-grained discrimination ability; Step 403: Construct interactive mining visual negative samples based on historical retrieval candidate images. The multimodal large language model is used to classify the Top-K candidate image sets output in each round of step 2 into three states: positive samples, simple negative samples, and difficult negative samples, and the historical erroneous retrieval results that violate the user's instructions are explicitly suppressed.

[0042] Specifically, step 5 consists of the following steps: Step 501: The three sets of difficult negative samples generated in step 4 are integrated through the dynamic difficult negative sample mining module to obtain dynamic difficult negative samples; Step 502: Calculate the dynamic contrastive loss based on global context embedding and dynamic hard negative samples, and calculate the progressive ranking constraint; Dynamic comparison loss The calculation formula is as follows:

[0043] in, Representing vectors with vector Temperature coefficient Scaled cosine similarity, for example Represents the current round's multimodal context embedding representation Embedded with target image Scaling cosine similarity; Represents the current round's multimodal context embedding representation Compared with the previous round of multimodal context embedding Scaling cosine similarity, Represents the current round's multimodal context embedding representation Compared with the previous round of decontextualized text query embedding Scaling cosine similarity; Represents the current round's multimodal context embedding representation With negative sample embedding The scaled cosine similarity, where, To mine visual negative sample sets from interactions With the in-class visual negative sample set Embed any negative sample taken from the set, For the first Multimodal context embedding representation of the wheel; For target image Embedded representation obtained by image encoder; and These are the multimodal context embedding and decontextualized text query embedding from the previous round, respectively; To interactively mine the visual negative sample set; It is the set of visual negative samples within the class.

[0044] Progressive sorting The calculation formula is as follows:

[0045] in, For indicator functions, when The value is 1 if the condition is met, and 0 otherwise. Constrain margins for sorting; Embed the cosine similarity with the target image for the current context; The cosine similarity between the previous round's context embedding and the same target image is used. The progressive ranking constraint is used to penalize feature degradation and stagnation during the retrieval process and to force the retrieval quality to monotonically increase across rounds.

[0046] Step 503: Construct a joint optimization objective by combining dynamic contrast loss and progressive ranking constraints. The student model parameters are iteratively optimized, and the student model is trained using the following formula:

[0047] in, To balance the hyperparameters; by minimizing The multi-step reasoning ability of the teacher model is distilled into the student model. The student model internalizes the multi-step reasoning ability of the teacher model in the feature space, and achieves accurate multi-round dynamic intent retrieval with a single forward propagation.

[0048] Specifically, step 6 includes the following: Step 601: In the offline stage, the trained student model is used to encode candidate images in the open-world image library to obtain the corresponding image embedding representation, and a vector index is constructed based on the image embedding representation; in one embodiment, the vector index is an approximate nearest neighbor index.

[0049] Step 602: During the online interaction phase, receive the text feedback from the user's current input and combine it with historical dialogue information and candidate images returned from the previous round to form the current round's multimodal dialogue context.

[0050] Step 603: Input the multimodal dialogue context into the trained student model, and extract the final hidden layer state of the special marker lexical at the end of the sequence as the current round context embedding representation.

[0051] Step 604: Based on the context embedding representation, perform similarity retrieval in the vector index to obtain and output Top-K candidate images that match the current retrieval intent.

[0052] Step 605: When the user continues to provide new feedback information on the returned candidate images, the new feedback information and the candidate images of the current round are incorporated into the multimodal dialogue context. Steps 603 and 604 are repeated to update the search results round by round until the target image that meets the user's needs is obtained.

[0053] In summary, this invention discloses a conversational image retrieval method for multi-turn dynamically evolving intents. This method is based on reasoning-guided embedding distillation and is used for user-initiated multi-turn interactive visual search. First, a user simulation framework based on a large model and a deep intent reasoning engine are constructed as the teacher model. Through structured thought chain reasoning, the multimodal dialogue history is reduced in dimensionality and purified into decontextualized text queries. Then, multi-turn dialogue training data containing both success and failure trajectories is synthesized through closed-loop interaction. Subsequently, the multi-step reasoning capability of the teacher model is implicitly transferred to a lightweight student model through the RIDE embedding distillation framework. Combined with three difficult negative sample mining mechanisms and progressive ranking constraints, collaborative optimization is performed at both semantic-level contrast alignment and trajectory-level dynamic evolution levels. Finally, multi-turn conversational image retrieval that balances retrieval accuracy, long-range intent tracking capability, and real-time response efficiency is achieved through a single forward propagation.

[0054] This invention achieves effective modeling and efficient retrieval of multi-turn dynamically evolving retrieval intentions by constructing a dynamic dialogue simulation framework, a deep intent reasoning engine, and a reasoning-guided embedding distillation framework. Among these, the dynamic dialogue simulation framework improves the authenticity and initiative of user feedback generation, the deep intent reasoning engine realizes decontextualized intent extraction from multi-turn multimodal contexts, and the three-layer progressive dynamic hard negative sample mining mechanism and progressive ranking constraints enhance the model's ability to distinguish long-term intent evolution, fine-grained differences, and historical error candidates. The processing method combining offline image library precoding and online single forward retrieval improves retrieval efficiency and engineering scalability in large-scale image library scenarios, thereby solving the problems of passive interaction methods, insufficient long-term intent tracking capabilities, and difficulty in balancing retrieval accuracy and real-time response efficiency in existing technologies.

[0055] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0056] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0057] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disks, optical discs, computer memory, random access memory, USB flash drives, portable hard drives, etc.) containing computer-usable program code.

[0058] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope disclosed in the present invention, based on the technical solution and inventive concept of the present invention, shall fall within the scope of protection of the present invention.

Claims

1. A conversational image retrieval method oriented towards multi-turn dynamic evolution intent, characterized in that, Includes the following steps: Step 1: Construct a multi-level classification system, build an open-world image library based on the multi-level classification system, and deploy a large-scale user simulation framework to generate multi-round conversational image retrieval training data. Step 2: Perform multi-round simulated retrieval interaction through the teacher model, generate decontextualized text queries based on the multimodal dialogue history to retrieve images from the library, and generate training data containing positive and difficult trajectories based on the retrieval results. Step 3: Build a RIDE embedding distillation framework with multiple functional modules as the student model, read the training trajectory output from Step 2, concatenate the multimodal input, and extract the global context embedding representing the retrieval intent. Step 4: Construct three types of difficult negative samples from three dimensions: historical interaction semantics, differentiated images of the same type in the image library, and candidate images from historical retrieval. Step 5: Construct a joint optimization objective using the three types of difficult negative samples and global context embedding, iteratively optimize the student model, and complete the knowledge distillation from the teacher model to the student model; Step 6: Offline pre-encode the images in the image library to build vector indexes, online receive multi-turn user feedback to splice multimodal dialogue context, use the trained student model to extract context embeddings and retrieve vector indexes, iterate interactively until the target image is output.

2. The conversational image retrieval method for multi-turn dynamic evolution intent according to claim 1, characterized in that, The specific steps of step 1 are as follows: Step 101: Construct a multi-level classification system, dividing visual concepts into five core dimensions from an ontological perspective: nature, human living environment, man-made objects, human society, and events and processes. Step 102: Organize each of the core dimensions into a four-layer progressive structure of macro layer, meso layer, micro layer and specific category layer, and construct an open world image library based on the multi-level classification system; Step 103: Deploy a user simulation framework based on a large model. Based on the open-world image library, generate multi-turn conversational image retrieval training data through structured intent planning, human-like realistic behavior modeling, and closed-loop interaction with state awareness.

3. The conversational image retrieval method for multi-turn dynamic evolution intent according to claim 2, characterized in that, Step 103 specifically includes: Step 10301: Perform structured intent planning on user feedback behavior and classify intent templates into six core interaction patterns; Step 10302: Model the simulated user’s real-world behavior. Generate conversational and fragmented natural language feedback through layperson role setting and a progressive disclosure mechanism. Each round will reveal a maximum of a preset number of new modification details. Step 10303: Perform state-aware closed-loop interaction. Before generating each round of feedback, analyze the difference between the current candidate image and the target image, verify the consistency between the modification points to be fed back and the historically confirmed features, and select the intent template accordingly. Step 10304: Generate feedback text based on the selected intent template, and evaluate the quality of simulated user feedback from the dimensions of visual fidelity, language naturalness, and behavioral consistency.

4. The conversational image retrieval method for multi-turn dynamic evolution intent according to claim 2, characterized in that, The specific steps of step 2 are as follows: Step 201: In the interactive round, the teacher model receives the multimodal dialogue history, which includes at least the Top-K candidate image set and text dialogue context of the previous round, and performs visual analysis and difference alignment on the multimodal dialogue history to form effective historical features; Step 202: Fuse the new visual constraints of the current round with the historical effective features to generate a decontextualized text query. Input the decontextualized text query into the visual-language retrieval device, retrieve the open-world image library for retrieval, and output the Top-K candidate image set for this round. If the target image is located in the Top-K candidate image set for this round, it is saved as a positive training trajectory. If it is not detected and the upper limit of the interaction round has not been reached, call the user simulation framework to generate new feedback and iterate the interaction. If no detection is found and the maximum number of rounds has been reached, the interaction trajectory is saved as a negative sample to construct a difficult trajectory.

5. The conversational image retrieval method for multi-turn dynamic evolution intent according to claim 4, characterized in that, Step 3 includes the following steps: Step 301: Build a student model based on the RIDE embedding distillation framework, which includes a multimodal context embedding generation module, a dynamic hard negative sample mining module, a dynamic contrastive loss calculation module, and a progressive ranking constraint module. Step 302: Read the training trajectory output in Step 2, extract the text dialogue context from the trajectory as the accumulated text retrieval requirements, extract the Top-K candidate images of each round as the recall images of the previous round, and concatenate the two to form a multimodal input; feed the multimodal input into the student model, and extract the hidden layer state of the special marker words at the end of the sequence as the global context embedding.

6. The conversational image retrieval method for multi-turn dynamic evolution intent according to claim 5, characterized in that, Step 4 includes the following steps: Step 401: Construct incomplete semantic negative samples based on historical interaction semantics, and use the previous round of multimodal context embedding and the previous round of decontextualized text query embedding as negative samples to participate in comparative learning, so as to force the model to encode the information increment of the current round; Step 402: Construct intra-class visual negative samples for differentiated images of the same type in the image library. Samples from the open-world image library that belong to the same semantic category as the target image but have differences in details are used to enhance fine-grained discrimination ability. Step 403: Construct interactive mining visual negative samples for historical retrieval candidate images. Use a multimodal large language model to divide the Top-K candidate image set output in each round of Step 2 into three-state classifications: positive samples, simple negative samples, and difficult negative samples. This is used to explicitly suppress historical erroneous retrieval results that violate user instructions.

7. The conversational image retrieval method for multi-turn dynamic evolution intent according to claim 6, characterized in that, The specific steps for step 5 are as follows: Step 501: Integrate the three sets of difficult negative samples generated in Step 4 through the dynamic difficult negative sample mining module to obtain dynamic difficult negative samples; Step 502: Calculate the dynamic contrastive loss based on global context embedding and dynamic hard negative samples, and calculate the progressive ranking constraint; Among them, dynamic contrast loss The calculation formula is as follows: ； in, For the first Multimodal context embedding representation of the wheel; For target image Embedded representation obtained by image encoder; and These are the multimodal context embedding and decontextualized text query embedding from the previous round, respectively; To interactively mine the visual negative sample set; The set of visual negative samples within the class; Represents the current round's multimodal context embedding representation Embedded with target image Scaling cosine similarity; Represents the current round's multimodal context embedding representation Compared with the previous round of multimodal context embedding Scaling cosine similarity, Represents the current round's multimodal context embedding representation Compared with the previous round of decontextualized text query embedding Scaling cosine similarity; Represents the current round's multimodal context embedding representation With negative sample embedding Scaling cosine similarity, To mine visual negative sample sets from interactions With the in-class visual negative sample set Embed any negative sample taken from the set; Step 503: Construct a joint optimization objective by combining the dynamic contrast loss and the progressive ranking constraint, and iteratively optimize the student model parameters.

8. A conversational image retrieval method for multi-turn dynamic evolution intent according to claim 7, characterized in that, The progressive sorting constraint in step 502 Defined as: for consecutive rounds and The current round requires multimodal context embedding. Embedded with target image The similarity relative to the previous round of context embedding Embedded with target image The similarity should at least increase the preset margin. The formula is as follows: ； in, For indicator functions, when The value is 1 if the condition is met; otherwise, the value is 0. Constrain margins for sorting; The cosine similarity between the current context embedding and the target image embedding; The progressive ranking constraint is used to force the similarity between the query feature and the target image to increase monotonically along the dialogue trajectory, based on the cosine similarity between the previous round of context embedding and the embedding of the same target image.

9. A conversational image retrieval method for multi-turn dynamic evolution intent according to claim 7, characterized in that, The joint optimization objective in step 503 is: ,in To balance the hyperparameters; by minimizing The multi-step reasoning ability of the teacher model is distilled into the student model.

10. A conversational image retrieval method for multi-turn dynamic evolution intent according to claim 1, characterized in that, Step 6 specifically includes: Step 601: In the offline stage, the trained student model is used to pre-encode candidate images in the open-world image library to obtain the image embedding representation corresponding to each image, and a vector index is constructed based on the image embedding representation. Step 602: During the online interaction phase, receive the user's current round of text feedback and the candidate images returned in the previous round, and combine them with historical dialogue information to form the current round of multimodal dialogue context. Step 603: Input the multimodal dialogue context into the trained student model, and extract the final hidden layer state of the special marker word at the end of the sequence as the current round context embedding representation; Step 604: Based on the context embedding representation, perform similarity retrieval in the vector index to obtain and output the Top-K candidate images that match the current retrieval intent; Step 605: When the user continues to input new feedback information, the new feedback information and the current round of candidate images are incorporated into the multimodal dialogue context, and steps 603 to 604 are repeated until the target image that meets the user's retrieval needs is output.

Images