Retrieval enhancement system and method based on multi-modal interaction agent

Abstract

The invention discloses a retrieval enhancement system and method based on a multi-modal interaction agent, and relates to the field of artificial intelligence, and the retrieval enhancement system comprises the following steps: obtaining an initial prompt; the initial prompt comprises first data of at least one mode; when a retrieval operation is determined to be performed on the basis of the initial prompt through a multi-modal interaction agent, retrieval is performed on the basis of the initial prompt through a retrieval model to obtain a retrieval result, and the retrieval result comprises second data of at least one modal; determining a target interaction model from a plurality of candidate interaction models through the multi-modal interaction agent based on the modal of the first data and the modal of the second data; and interacting with a user based on the initial prompt and the retrieval result through the target interaction model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a retrieval enhancement system and method based on multimodal interactive intelligent agents. Background Technology

[0002] Question answering systems, as an important application of information retrieval and natural language processing technologies, typically consist of two core modules: a retrieval system and a reader. This system can accurately retrieve and generate natural language answers from massive amounts of data.

[0003] After obtaining search results through a question-and-answer system, users need to interact with the system to obtain the final answer or further information. However, existing technologies typically employ fixed interaction models, which struggle to meet the diverse needs of users for interaction methods and quality in different scenarios.

[0004] Therefore, improving the accuracy of interaction is an urgent problem to be solved. Summary of the Invention

[0005] This application provides a retrieval enhancement system and method based on multimodal interactive intelligent agents to improve the accuracy of interaction.

[0006] In a first aspect, embodiments of this application provide a retrieval enhancement system based on a multimodal interactive intelligent agent, the retrieval enhancement system being used for:

[0007] Obtain an initial prompt; the initial prompt includes first data of at least one modality;

[0008] When a retrieval operation is determined based on the initial prompt by a multimodal interactive intelligent agent, a retrieval model is used to perform a retrieval based on the initial prompt to obtain retrieval results, which include: second data in at least one modality;

[0009] The multimodal interactive agent determines a target interactive model from multiple candidate interactive models based on the modality of the first data and the modality of the second data; and interacts with the user based on the initial prompts and the search results using the target interactive model.

[0010] In one possible implementation, the plurality of candidate interaction models includes: a vertical domain multimodal interaction model, and a vertical domain large language model. The determination of the target interaction model from the plurality of candidate interaction models based on the modality of the first data and the modality of the second data includes:

[0011] In response to the fact that the modality of the first data is text modality and the modality of the second data is text modality, the vertical domain large language model is determined to be the target interaction model;

[0012] or,

[0013] In response to the first data including a non-textual modality, and / or the second data including a non-textual modality, the vertical multimodal interaction model is determined as the target interaction model.

[0014] In one possible implementation, the retrieval enhancement system is further used for:

[0015] Through the multimodal interactive agent, in response to the initial prompt, it is determined that no retrieval operation will be performed, and the modality of the first data is text modality, and the vertical domain large language model is determined as the target interaction model;

[0016] or,

[0017] Through the multimodal interactive agent, in response to the initial prompt, it is determined that no retrieval operation will be performed, and the modality of the first data includes the non-text modality, thus determining the vertical domain multimodal interaction model as the target interaction model.

[0018] In one possible implementation, the retrieval enhancement system is further used for:

[0019] Receive the user-triggered interaction model switching instruction; the interaction model switching instruction is used to instruct the target interaction model to be switched to the vertical domain large language model;

[0020] In response to the interaction model switching command, the multimodal interactive agent outputs a prompt message when the modality of the first data includes a non-text modality and / or the modality of the second data includes a non-text modality. The prompt message is used to instruct the vertical domain large language model not to reference the non-text modality data.

[0021] In one possible implementation, the retrieval enhancement system is further used for:

[0022] Obtain a vertical multimodal sample dataset, which includes: vertical multimodal sample cues and vertical sample responses;

[0023] The preset model is trained using the vertical multimodal sample dataset to obtain the vertical multimodal interaction model.

[0024] The preset model includes: a hidden layer, a text output module, and a multimodal feature vector output module; the hidden layer is used to extract semantic features based on the multimodal sample prompts to obtain semantic features that characterize the multimodal sample prompts; the text output module is used to output predicted text describing the multimodal sample prompts based on the semantic features; the multimodal feature vector output module is used to output predicted multimodal features characterizing the multimodal sample prompts based on the semantic features; the predicted text, the predicted multimodal features, and the vertical sample responses are used to train the preset model.

[0025] In one possible implementation, training a preset model using the vertical multimodal sample dataset to obtain the vertical multimodal interaction model includes:

[0026] The trainable parameters of the multimodal feature vector output module are frozen, and a single-round dialogue training is performed on the part of the preset model other than the multimodal feature vector output module using the vertical multimodal sample dataset to obtain the first initial vertical multimodal interaction model.

[0027] The trainable parameters of the multimodal feature vector output module are frozen, and the parts of the first initial vertical multimodal interaction model other than the multimodal feature vector output module are trained through multiple rounds of dialogue using the vertical multimodal sample dataset to obtain the second initial multimodal reasoning large model.

[0028] The trainable parameters of the multimodal feature vector output module are frozen, and reinforcement learning training is performed on the part of the second initial multimodal inference large model other than the multimodal feature vector output module using the vertical multimodal sample preference dataset to obtain the multimodal inference large model.

[0029] In one possible implementation, the interaction with the user based on the initial prompt and the search results through the target interaction model includes:

[0030] Through the target interaction model, based on the initial prompts and the search results, multiple rounds of interaction are conducted with the user;

[0031] Any round of interaction in the multi-round interaction process includes at least the following steps:

[0032] Step 1: Through the target interaction model, in response to the interaction information of this round of interaction based on the user input, determine to perform a retrieval operation, call the retrieval model, and perform a retrieval based at least on the interaction information to obtain new retrieval results;

[0033] Step 2: Based on the new search results, a new target interaction model is determined from the multiple candidate interaction models; the new target interaction model is used to conduct the next round of interaction with the user based on the interaction information and the new search results.

[0034] Secondly, embodiments of this application provide a retrieval enhancement method based on a multimodal interactive intelligent agent, the method comprising:

[0035] Obtain an initial prompt; the initial prompt includes first data of at least one modality;

[0036] When a retrieval operation is determined based on the initial prompt by a multimodal interactive intelligent agent, a retrieval model is used to perform a retrieval based on the initial prompt to obtain retrieval results, which include: second data in at least one modality;

[0037] The multimodal interactive agent determines a target interactive model from multiple candidate interactive models based on the modality of the first data and the modality of the second data; and interacts with the user based on the initial prompts and the search results using the target interactive model.

[0038] Thirdly, embodiments of this application provide a retrieval enhancement device based on a multimodal interactive intelligent agent, comprising:

[0039] An acquisition module is used to acquire an initial prompt; the initial prompt includes first data of at least one modality;

[0040] The retrieval module is used to perform a retrieval based on the initial prompt when a retrieval operation is determined to be performed through a multimodal interactive intelligent agent, and to obtain retrieval results, wherein the retrieval results include: second data of at least one modality;

[0041] An interaction module is used to determine a target interaction model from multiple candidate interaction models based on the modality of the first data and the modality of the second data through the multimodal interactive agent; and to interact with the user through the target interaction model based on the initial prompts and the search results.

[0042] Fourthly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0043] The memory stores computer-executed instructions;

[0044] The processor executes computer execution instructions stored in the memory, causing the processor to perform the method described in the second aspect above.

[0045] Fifthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in the second aspect above.

[0046] In a sixth aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the second aspect above.

[0047] This application provides a retrieval enhancement system and method based on a multimodal interactive intelligent agent. First, based on initial prompts, a retrieval operation is determined, and retrieval results are obtained through a retrieval model. Second, by combining the modality of the initial prompts and / or retrieval result data, a target interaction model is determined from multiple candidate interaction models, improving the targeting and accuracy of the interaction. Furthermore, this application embodiment can engage in interaction with the user based on the initial prompts and / or retrieval results, enhancing the accuracy of the interaction. Attached Figure Description

[0048] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A schematic diagram of a retrieval enhancement system based on a multimodal interactive intelligent agent provided in an embodiment of this application;

[0051] Figure 2 A flowchart illustrating a method for obtaining image-text pairs provided in an embodiment of this application;

[0052] Figure 3 A schematic diagram of the model architecture of a preset model provided in an embodiment of this application;

[0053] Figure 4 A flowchart illustrating a method for training a preset model provided in an embodiment of this application;

[0054] Figure 5 A schematic diagram illustrating the training process of a large multimodal inference model provided in the application embodiment;

[0055] Figure 6 A schematic diagram of the architecture of a retrieval model provided in an embodiment of this application;

[0056] Figure 7 A schematic diagram illustrating the training process of the multimodal embedding model provided in the embodiments of this application;

[0057] Figure 8 A schematic diagram illustrating the construction of a multimodal database and domain knowledge graph, provided as an embodiment of this application;

[0058] Figure 9 A schematic diagram illustrating a process for constructing a domain knowledge graph, provided as an embodiment of this application;

[0059] Figure 10 A schematic diagram illustrating a retrieval enhancement method based on a multimodal interactive intelligent agent provided in an embodiment of this application;

[0060] Figure 11 A schematic diagram of the structure of a retrieval enhancement device based on a multimodal interactive intelligent agent provided in this application;

[0061] Figure 12 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0062] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0063] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0064] A question-answering system consists of two parts: a retrieval module and a reader. The retrieval module retrieves relevant candidate fragments from a large corpus based on the user's question. These fragments contain the information needed to answer the user's question. Currently, commonly used retrieval modules include sparse representation retrieval and dense vector retrieval. The reader module then refines, infers, and summarizes the retrieved information, ultimately providing a human-like (natural language) answer.

[0065] Common readers employ the Transformer (a deep learning model) architecture and are generally divided into two types: information extraction readers, such as BERT (a bidirectional encoder technique) and RoBERTa (a pre-training method), and text-to-text generative readers, such as T5 (a text-to-text transformation transformer), BART (a bidirectional autoregressive transformer), and GPT (a generative pre-trained transformer).

[0066] The role of a reader is to answer user questions by combining search results. Multimodal large models, benefiting from their large number of parameters and training on massive amounts of data, possess powerful understanding and summarization capabilities. With the development of large model technology and the improvement of application effects, multimodal large models are now widely used in readers.

[0067] Search engines can be categorized into several main types, including same-modal search, cross-modal search, and multimodal-to-multimodal search. The search process typically begins by using an embedding model to extract embedding vectors for all data to be retrieved. During the search, the input query is used to extract a query embedding vector using the same embedding model, which is then compared with the embedding vectors of all data to be retrieved to obtain the candidates with the highest similarity.

[0068] Currently, question-answering systems have evolved to the Retrieval-Augmented Generation (RAG) stage, and continue to evolve into more powerful forms such as Advanced RAG, Modular RAG, Agentic RAG, and Multimodal RAG.

[0069] After obtaining search results through a question-and-answer system, users need to interact with the system to obtain the final answer or further information. However, existing technologies typically employ fixed interaction models, which struggle to meet the diverse needs of users for interaction methods and quality in different scenarios.

[0070] For example, fixed interaction models often limit input methods to a few common forms, such as supporting only plain text input. This single input method greatly restricts the user's ability to express their questions. On the output side, fixed interaction models usually use standardized text formats for responses, making it difficult to present them through charts, videos, or other means.

[0071] Therefore, embodiments of this application provide a retrieval enhancement system and method based on a multimodal interactive intelligent agent. First, a retrieval operation is determined based on initial prompts, and retrieval results are obtained through a retrieval model. Second, by combining the modality of the initial prompts and / or retrieval result data, a target interaction model is determined from multiple candidate interaction models, which improves the targeting and accuracy of the interaction. Furthermore, embodiments of this application can engage in interaction with the user based on initial prompts and / or retrieval results, enhancing the accuracy of the interaction.

[0072] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0073] Figure 1 This is a schematic diagram of a retrieval enhancement system based on a multimodal interactive intelligent agent, provided as an embodiment of this application. For example... Figure 1 As shown, the retrieval enhancement system includes:

[0074] S101. Obtain initial prompts, the initial prompts including first data of at least one modality.

[0075] Optionally, the initial prompt can be the user's input, such as a question or description of a need, which serves as the basis for subsequent retrieval and interaction. The modality can be the type of primary data; different modalities can represent different types of information. For example, a modality can be any one or more of the following: text modality, image modality, etc.

[0076] For example, in the field of emergency safety, the initial prompt can be a textual description related to the field of emergency safety, and / or visual materials related to the field of emergency safety. For example, a 10-second clip of an anomaly or emergency event can be extracted at 1-3 frames per second, and the resulting multiple images of the anomaly or emergency event can be used to form the initial prompt.

[0077] Optionally, the multimodal interactive agent can be configured with, for example, an Application Programming Interface (API) or a Graphical User Interface (GUI). The multimodal interactive agent can obtain the initial prompts from the user input through the aforementioned API or GUI.

[0078] S102. Through a multimodal interactive intelligent agent, when a retrieval operation is determined based on the initial prompt, a retrieval is performed based on the initial prompt through a retrieval model to obtain retrieval results, which include: second data of at least one modality.

[0079] Optionally, the retrieval model may be able to understand the meaning of the initial prompts, convert them into computer-processable query instructions, and search for relevant information according to preset rules and algorithms to perform the retrieval operation. Consistent with the first data, the modality of the second data can also be any one or more of the following: text modality, image modality, etc.

[0080] Optionally, the multimodal interactive agent may include, for example, any existing large language model. The multimodal interactive agent can construct a prompt word based on the initial prompt (this prompt word can be used to instruct the analysis of the initial prompt to determine whether a retrieval is required in response to the initial prompt), and input the prompt word into the large language model to obtain a judgment result indicating whether a retrieval operation is required.

[0081] For example, if a multimodal interactive agent can directly answer the requests and / or implicit requests in the initial prompts based on its own database, no retrieval operation is required; however, if the multimodal interactive agent needs to call a retrieval model to answer the requests and / or implicit requests in the initial prompts, then a retrieval operation is required.

[0082] S103. Using a multimodal interactive agent, a target interactive model is determined from multiple candidate interactive models based on the modality of the first data and the modality of the second data; and the target interactive model is used to interact with the user based on the initial prompts and search results.

[0083] Optionally, multiple candidate interaction models can each have their own interaction logic, processing methods, and applicable scenarios. For example, some candidate interaction models are good at handling text-to-text interactions, such as text-based question-and-answer interactions; some candidate interaction models are more suitable for handling interaction scenarios that combine text and images, such as performing comprehensive analysis and responses based on user-input text descriptions and uploaded images; and some candidate interaction models can handle multimodal input and output, such as simultaneously processing text and outputting images as input and output.

[0084] A multimodal interactive agent can, for example, determine the modality of first data by reading its format identifier. Similarly, a multimodal interactive agent can, for example, determine the modality of second data by reading its format identifier. Optionally, the multimodal interactive agent can, for example, store a mapping relationship between data modalities and candidate interaction models. Based on the modality of the first data, the modality of the second data, and the mapping relationship between the data modalities and candidate interaction models, the multimodal interactive agent can determine a target interaction model from multiple candidate interaction models.

[0085] Optionally, interaction with the user can be a process of information exchange and interaction between the two parties. The multimodal interactive agent can receive initial prompts from the user through a target interaction model, analyze and process the initial prompts and search results using the target interaction model, and then generate corresponding responses to provide feedback to the user. For example, this target interaction model can refer to any existing large language model. The user's feedback may then trigger further interactive operations by the multimodal interactive agent, forming an interaction loop.

[0086] Optionally, the interaction process can be a single-turn interaction or a multi-turn interaction. Taking multi-turn interaction as an example, any turn of the interaction process in a multi-turn interaction includes at least the following steps:

[0087] Step 1: Through the target interaction model, in response to the interaction information of this round of interaction based on user input, determine to perform a retrieval operation, call the retrieval model, and perform a retrieval based at least on the interaction information to obtain new retrieval results.

[0088] Optionally, the interaction information is similar to the initial prompt and may include data from at least one modality. Based on the interaction information, the multimodal interactive agent determines to perform a retrieval operation. The process of invoking the retrieval model is similar to step S102 above and will not be repeated here.

[0089] The retrieval enhancement system can perform retrieval based on interaction information to obtain new retrieval results. The retrieval enhancement system can also perform retrieval based on interaction information from the current round of interaction, interaction information from previous rounds, and initial prompts. This application does not limit this aspect.

[0090] Step 2: Based on the new search results, a new target interaction model is determined from multiple candidate interaction models. The new target interaction model is used to conduct the next round of interaction with the user based on the interaction information and the new search results.

[0091] Optionally, the retrieval enhancement system can redetermine a new target interaction model from multiple candidate interaction models based on the new modality of the retrieval results.

[0092] This application's embodiments utilize a multi-round interaction mechanism to dynamically adjust the retrieval strategy and interaction model based on real-time user input. Each round of interaction is optimized based on the results of the previous round, ensuring the continuity and relevance of the interaction, and improving the speed and quality of responding to user searches.

[0093] This application embodiment can first determine the search operation based on initial prompts and obtain search results through a search model. Secondly, by combining the modality of the initial prompts and / or search result data, a target interaction model is determined from multiple candidate interaction models, which can improve the targeting and accuracy of the interaction. Furthermore, this application embodiment can engage in interaction with the user based on the initial prompts and / or search results, enhancing the accuracy of the interaction.

[0094] Taking multiple candidate interaction models, including vertical domain multimodal interaction models and vertical domain large language models, as examples, the following section provides a detailed explanation of how a multimodal interaction agent determines the target interaction model from multiple candidate interaction models based on the modality of the first data and the modality of the second data.

[0095] When both the first and second data are text-based, the multimodal interactive agent can identify the vertical domain large language model as the target interaction model.

[0096] Optionally, the vertical domain large language model can be a large language model capable of in-depth analysis and processing of text data within a vertical domain, providing text interaction services. For example, the vertical domain large language model can be trained based on vertical domain data, building upon a large language model. The vertical domain can be a specific vertical field, such as the emergency safety field.

[0097] When the modality of the first data includes a non-textual modality, and / or the modality of the second data includes a non-textual modality, the multimodal interactive agent can determine the vertical multimodal interaction model as the target interaction model.

[0098] Optionally, a vertical multimodal interaction model can be a model that can process and integrate multiple different types of data modalities, such as text, images, and audio, in a vertical domain, and analyze and interact with inputs containing multiple modal data.

[0099] In one implementation, the multimodal interactive agent can acquire a vertical multimodal sample dataset, which includes: vertical multimodal sample cues and vertical sample responses.

[0100] Optionally, the vertical-domain multimodal sample prompt and the vertical-domain sample response can be image-text pairs, where the image in the image-text pair corresponds to the vertical-domain sample response, and the text in the image-text pair corresponds to the vertical-domain multimodal sample prompt. The multimodal interactive agent can, for example, be configured with an API to access an open-source database and directly obtain the image-text pair from it, or it can process the image, table, or video to obtain the image-text pair.

[0101] Taking the example of a multimodal interactive intelligent agent processing images, tables, and videos to obtain image-text pairs, Figure 2 This is a flowchart illustrating a method for obtaining image-text pairs provided in an embodiment of this application, as shown below. Figure 2 As shown, the method for obtaining image-text pairs is as follows:

[0102] 1. Use the existing multimodal large model to process all the images, tables and videos to obtain the corresponding detailed text descriptions, that is, obtain image-text pairs.

[0103] 2. Use another existing multimodal large model to score the text description matching degree of the image-text pairs generated in step 1, and retain image-text pairs with a text description matching degree higher than the threshold. For example, retain image-text pairs with a text description matching degree higher than 8.

[0104] 3. Use existing large language models to analyze the text descriptions in all image-text pairs and filter out image-text pairs that do not match the target domain.

[0105] 4. Use the visual encoder in the existing multimodal large model as the feature extractor for image data, extract the representation vectors only for the image data in all image-text pairs, and store the representation vectors of these images in the vector database.

[0106] 5. Filter vectors in the vector database based on similarity. For example, only one image is kept for each representation vector with a similarity greater than 0.95, and 50% of the images with a similarity between 0.85 and 0.95 are randomly removed, resulting in high-quality, non-repeating image-text pairs for the target domain.

[0107] After acquiring the vertical multimodal sample dataset, the retrieval enhancement system can also be used to train a pre-defined model to obtain a vertical multimodal interaction model. This pre-defined model may include a hidden layer, a text output module, and a multimodal feature vector output module. Figure 3 This is a schematic diagram of a preset model provided in an embodiment of this application, such as... Figure 3As shown, the preset model may include image preprocessing, a visual encoder, such as a Vision Transformer (a model architecture for image processing), pixel unshuffle, a projector, a prompt, a text tokenizer, a Large Language Model (LLM), an LM Head, an Embedding Projector, an Output Text module, and a Multi-vectors module.

[0108] The preset model may also include a hidden layer. Figure 3 The hidden layer is not shown in the diagram. It is used to extract semantic features based on vertical multimodal sample cues and vertical sample responses, resulting in semantic features that characterize the vertical multimodal sample cues and vertical sample responses.

[0109] The text output module can be used to output predicted text based on semantic features, which can be used to describe the vertical multimodal sample prompts and vertical sample responses.

[0110] Optionally, the text output module can gain a deeper understanding of the relationship between vertical multimodal sample prompts and vertical sample responses based on semantic features, and then generate and output predicted text describing the vertical multimodal sample prompts and vertical sample responses. This predicted text can be used to train a preset model.

[0111] The multimodal feature vector output module is used to output predicted multimodal features based on semantic features, which characterize the vertical multimodal sample cues and vertical sample responses.

[0112] Optionally, the multimodal feature vector output module can fuse the semantic features of vertical multimodal sample cues and vertical sample responses, extract key semantic elements from them, and obtain predicted multimodal features based on the mapping relationship between key semantic elements and multimodal feature vectors. These predicted multimodal features can then be used to train a pre-defined model.

[0113] This application embodiment constructs a vertical multimodal sample dataset and trains a preset model to obtain a vertical multimodal interaction model that simultaneously processes text and multimodal features. This model has powerful semantic understanding capabilities and can also output multimodal feature vectors, providing a rich information foundation for subsequent interactions and improving the accuracy and diversity of interactions.

[0114] Figure 4 This is a flowchart illustrating a method for training a preset model provided in an embodiment of this application, as shown below. Figure 4As shown, the training process of the above-mentioned preset model can be as follows:

[0115] S201. Freeze the trainable parameters of the multimodal feature vector output module, and use the vertical multimodal sample dataset to perform single-round dialogue training on the part of the preset model other than the multimodal feature vector output module to obtain the first initial vertical multimodal interaction model.

[0116] Optionally, trainable parameters can be parameters in the pre-defined model that can be adjusted through optimization algorithms during training. A single-turn dialogue involves only one interaction between the user and the pre-defined model; the user asks a question or gives a command, and the pre-defined model generates a corresponding response based on the input. Single-turn dialogue training teaches the pre-defined model how to generate accurate, reasonable, and context-appropriate responses based on the input single-turn dialogue data.

[0117] The first initial vertical domain multimodal interaction model can be a model obtained by performing single-round dialogue training on the part of the preset model except for the multimodal feature vector output module. This first initial vertical domain multimodal interaction model is the product of preliminary training in a vertical domain multimodal environment, and has a certain vertical domain single-round dialogue interaction capability, and can handle simple multimodal interaction tasks in the vertical domain.

[0118] S202. Freeze the trainable parameters of the multimodal feature vector output module, and use the vertical multimodal sample dataset to perform multi-round dialogue training on the part of the first initial vertical multimodal interaction model other than the multimodal feature vector output module, to obtain the second initial multimodal reasoning large model.

[0119] Optionally, multi-turn dialogue can involve multiple interactions between the user and the initial multimodal interaction model (IMM). After the user responds to a question or instruction given in the previous turn, the user may ask further questions or provide more information based on the response. The IMM needs to combine contextual information to provide subsequent responses. Multi-turn dialogue training aims to teach the model how to accurately understand the user's intent based on context and generate coherent and reasonable responses in such dialogues with multiple rounds of interaction.

[0120] Compared to the first initial model, the second initial multimodal reasoning model has stronger capabilities in handling multi-turn dialogues and performing reasoning. It can better adapt to complex multi-turn interaction scenarios within a vertical domain, gain a deeper understanding of user needs, and provide reasonable reasoning results.

[0121] S203. Freeze the trainable parameters of the multimodal feature vector output module, and use the vertical multimodal sample preference dataset to perform reinforcement learning training on the part of the second initial multimodal inference large model other than the multimodal feature vector output module to obtain the multimodal inference large model.

[0122] Optionally, the vertical domain multimodal sample preference dataset can be a sample set of data from multiple modalities and corresponding preference information. Reinforcement learning allows an agent to learn through interaction with its environment, optimizing its behavioral strategies by continuously trying and receiving feedback. During reinforcement learning training, the agent selects an action based on its current state, and the environment provides a reward or penalty signal based on this action. The agent then adjusts its strategy based on this signal to maximize long-term cumulative rewards.

[0123] A large-scale multimodal reasoning model can be obtained by training a second initial large-scale multimodal reasoning model through reinforcement learning. It possesses powerful multimodal data processing and reasoning capabilities within a specific vertical domain, and can integrate information from multiple modalities to provide accurate, reasonable, and user-expected reasoning results based on user input and preferences.

[0124] For example, Figure 5 A schematic diagram illustrating the training process of a large multimodal inference model provided in the application embodiments, as shown below. Figure 5 As shown, the training process of this large-scale multimodal inference model is as follows:

[0125] 1. Freeze the Embedding task and train the LM Head (language model head) only on the vertical multimodal dataset through single-turn dialogue.

[0126] 2. Freeze the Embedding task and train the LM Head only on the vertical multimodal dataset through multi-turn dialogue.

[0127] 3. Freeze the Embedding task and train the LM Head only on the vertical multimodal dataset (i.e., the vertical multimodal sample preference dataset) through reinforcement learning.

[0128] Optionally, the vertical multimodal sample preference dataset is a reconstruction of the vertical multimodal sample dataset, and the reconstruction method can refer to the existing implementation method.

[0129] This application's embodiments employ a phased training strategy to progressively optimize the performance of a vertical domain multimodal interaction model. This phased training strategy first performs single-turn dialogue training to equip the model with basic interactive capabilities. Subsequently, multi-turn dialogue training enhances the model's contextual understanding and continuous interaction capabilities. Finally, reinforcement learning training further improves the model's reasoning and decision-making abilities, effectively enhancing the model's interaction quality and stability.

[0130] This application's embodiments select a suitable interaction model based on the modality types of the first and second data. When only text is involved, a vertical domain large language model is selected for efficient processing; when non-text modalities are involved, a vertical domain multimodal interaction model is switched to ensure a comprehensive understanding and accurate response to multimodal information, thereby improving the accuracy of the interaction.

[0131] The above embodiments illustrate the relevant content of a multimodal interactive agent determining to perform a search operation in response to an initial prompt. The following will introduce the relevant content of a multimodal interactive agent determining not to perform a search operation in response to an initial prompt.

[0132] The retrieval enhancement system is also used to determine the target interaction model by using a multimodal interactive agent in response to an initial prompt, where no retrieval operation is performed. If the modality of the first data is text-based, the target interaction model can be identified as the vertical domain large language model. If the modality of the first data includes non-text-based modalities, the target interaction model can be identified as the vertical domain multimodal interaction model.

[0133] Optionally, the multimodal interactive agent may include, for example, any existing large language model. The multimodal interactive agent can construct a prompt word based on the initial prompt (this prompt word can be used to indicate whether the initial prompt needs to be analyzed to determine whether a retrieval is required in response), and input the prompt word into the large language model to obtain a judgment result indicating whether a retrieval operation is required. When this judgment result indicates that a retrieval is not required, the multimodal interactive agent can determine that no retrieval operation should be performed.

[0134] For example, when the user input contains an image or the search results contain an image, the vertical multimodal interaction model is selected to answer the user's question; when the user input does not contain an image or the search results do not contain an image, the vertical large language model is selected to answer the user's question.

[0135] Optionally, the vertical domain large language model and the vertical domain multimodal interaction model are similar to those described above, and will not be repeated here.

[0136] This application provides a model selection strategy for situations where there is no retrieval requirement. The interaction model can be determined based on the modality type of the initial prompt. For plain text input, a vertical domain large language model is used for a fast response; for input containing non-text modalities, a vertical domain multimodal interaction model is enabled, ensuring accurate interaction in different scenarios.

[0137] After the multimodal interactive agent determines the target interaction model, the multimodal interactive agent can be configured with a GUI to receive user-triggered interaction model switching instructions. The interaction model switching instructions are used to instruct the target interaction model to be switched to the vertical domain large language model.

[0138] Optionally, the user-triggered interaction model switching command can be triggered through the interface of the multimodal interactive agent interacting with the user. For example, the switching command can be any one or more of the following: voice command, text input, gesture operation, etc.

[0139] The multimodal interactive agent can be configured with a GUI that, in response to an interaction model switching command, outputs a prompt message through the GUI when the modality of the first data includes a non-textual modality and / or the modality of the second data includes a non-textual modality. The prompt message is used to instruct the vertical domain large language model not to reference the non-textual modal data.

[0140] Optionally, the multimodal interactive agent can be configured with a GUI. In response to an interaction model switching command, when the modality of the first data includes a non-text modality, the GUI outputs a prompt message indicating that the vertical domain large language model should not reference non-text modal data. The multimodal interactive agent can also respond to an interaction model switching command when the modality of the second data includes a non-text modality, and further respond to both instances where the modality of the first and second data includes a non-text modality.

[0141] Optionally, the multimodal interactive agent can output prompts via pop-up windows. For example, this prompt could be, "Please note that the vertical domain large language model cannot reference images or videos from the conversation history." The vertical domain large language model can then place non-textual modal data as placeholders before continuing its response.

[0142] This application's embodiments introduce model switching instructions and corresponding prompts. When a user attempts to switch the interaction model to a vertical large language model that is not suitable for the current non-text modality data, the system will provide clear prompts, avoiding invalid or erroneous interactions and improving the accuracy of the interaction.

[0143] Based on the above embodiments, Figure 6 This is a schematic diagram of the architecture of a retrieval model provided in an embodiment of this application, such as... Figure 6 As shown, the retrieval model can respond to user input text and / or input images. The input text is rewritten into multiple queries by the Query (query) and intent recognition model, such as query1 and query2.

[0144] For example, the user's input text could be "Check if the current monitoring system has captured any abnormal behaviors such as horseplay, smoking, or falling down." The output of the rewritten retrieval model would be: query1: people smoking, query2: people fighting, query3: climbing over a wall.

[0145] The rewritten input text and / or the input image are then input into the multimodal embedding model, the architecture of which can be referenced above. Figure 3 It can include a text tokenizer and ViT (a model architecture for image processing). Figure 7 A schematic diagram illustrating the training process of the multimodal embedding model provided in this application embodiment, as shown below. Figure 7 As shown, the training process for a multimodal embedding model can be achieved through the following steps:

[0146] 1. Freeze the Embedding task and train the LM Head (language model head) only on the vertical multimodal dataset. It should be noted that the vertical multimodal dataset can be a dataset of vertical multimodal samples.

[0147] 2. Freeze the text generation task and train the Embedding task only on the vertical multimodal dataset.

[0148] 3. For the same batch of training data, there is a 50% chance of randomly starting either the generation or embedding task, and a 50% chance of performing both tasks simultaneously. This training data can include a multimodal dataset from a specific industry and an Optical Character Recognition (OCR) dataset. The multimodal dataset can account for up to 70%, and the OCR dataset can account for up to 30%.

[0149] 4. All batches of training data are used simultaneously for two tasks, but only one batch (epoch) is executed in this stage. This training data is consistent with the training data in step 3.

[0150] The multimodal embedding model can provide cross-modal retrieval services based on the Model Context Protocol (MCP), including: basic semantic retrieval, cross-modal semantic retrieval, sparse retrieval / keyword retrieval, and hybrid retrieval. This cross-modal retrieval service can perform searches based on multimodal databases and domain knowledge graphs to ultimately obtain search results.

[0151] There are two approaches to handling multimodal problems in current multimodal RAGs. One approach is to convert all modal inputs other than text into text and then use a large language model for retrieval enhancement and generation. The other approach is to use a multimodal embedding model to retrieve the text questions and images input by the user, and then pass the retrieval results to a large multimodal model for multi-turn dialogue.

[0152] The first method has two drawbacks: First, it is highly dependent on the results of converting other modalities to text modalities, which can lead to the loss of some information and result in unsatisfactory retrieval enhancement output. Second, the process of converting other modalities to text modalities requires a large multimodal model with a large number of parameters, resulting in a high computational cost. While the second method avoids information loss, traditional multimodal embedding models can only convert input image or text information into embedding vectors. They cannot perform sparse or mixed retrieval on modal information other than text.

[0153] Therefore, the retrieval model provided in this application embodiment, compared with the multimodal RAG in the prior art, can realize cross-modal retrieval (such as text-to-image search, image-to-text search, or mixed retrieval), cover complex query scenarios, and realize multimodal retrieval.

[0154] The following section provides a detailed explanation of how to construct a multimodal database and domain knowledge graph. Figure 8 This application provides a schematic diagram illustrating the construction of a multimodal database and a domain knowledge graph, as shown in the embodiments of this application. Figure 8 As shown, images and videos can be used to obtain a multimodal database based on a multimodal embedding model and system prompts. This multimodal database can include multi-vectors (multimodal feature vectors) and text descriptions. Image-text documents and text documents can be used to construct a domain knowledge graph based on a rich text document parsing module and a multimodal embedding model.

[0155] Rich text documents, in particular, can be document types that, in addition to text content, can also contain rich formatting and elements such as font styles, images, tables, charts, audio and video, and hyperlinks. For the rich text document parsing module, Figure 9 This application provides a schematic diagram of a process for constructing a domain knowledge graph, as illustrated in the embodiments of this application. Figure 9 As shown, image and text documents and text documents can include Portable Document Format (PDF), Office (a document format), and plain text documents.

[0156] The multimodal interactive agent can be configured with a conversion tool to convert documents to PDF format. This tool can convert all the aforementioned domain knowledge documents into PDF format, and then into images in Joint Photographic Experts Group (JPG) format. The layout element extraction model can identify layout elements in the images, extract element numbers, and segment them into blocks based on areas such as images, tables, body text, and titles.

[0157] Optionally, the layout element extraction model can automatically identify and extract various layout elements from the image converted from the rich text document. This model can employ existing deep learning models. For example, the layout elements may include one or more of the following: figures, tables, chapter names, text paragraphs, figure / table titles, figure / table captions, etc. The attribute information of the layout elements can be information describing their characteristics and properties; for example, the attribute information may include the layout element number.

[0158] Optionally, the layout element extraction model can first perform multiple batch processing operations on all the images converted from rich text documents until all the converted images are processed. Secondly, based on the recognition results of each page, the model can segment out the regions of all elements on all pages, obtaining images corresponding to all elements in all rich text documents and all pages. Thirdly, the model can divide the images corresponding to all elements into two main categories, corresponding to different system prompts: text elements and chart elements. Finally, the model can group elements by image size within each category, reducing size differences within the same batch and avoiding unnecessary computational increases caused by alignment issues.

[0159] The multimodal large model (i.e., the multimodal embedding model mentioned above) can extract text and knowledge from each element by batch processing text elements and graph / table elements. The multimodal large model (i.e., the multimodal embedding model mentioned above) can extract embedding vectors from the sliced ​​text, including recombining the recognition results of each element into a Markdown (a markup language) plain text document, and extracting vectors based on document slices. Optionally, the multimodal interactive agent may include a document generation and writing module. Through this module, the text information corresponding to all elements is written into the document in a top-to-bottom, left-to-right order, resulting in a text document corresponding to the original rich text document.

[0160] In summary, multimodal interactive intelligent agents can construct domain knowledge graphs based on any existing knowledge graph construction logic, such as page element numbers, document slice vector extraction, and document metadata.

[0161] In one implementation, the domain knowledge graph can include six entity types and six relation types, as shown below:

[0162] Domain (Emergency Safety Sub-sectors): Gas, Water, Petrochemical, Transportation, etc.

[0163] Doc Category: Document Type 1, Document Type 2, Document Type 3, etc.

[0164] Document: papers, regulations, management rules, etc.

[0165] Page: page1, page2, page3, etc.

[0166] Content Element: Image elements, table elements, heading elements, etc.

[0167] Knowledge Chunk: Document Chunk 1, Document Chunk 2, Document Chunk 3, etc.

[0168] The page entity contains four attributes: original document identifier (ID), number of images, number of tables, and page overview. The element entity contains five attributes: original document ID, original page ID, save path, text information, and representation vector. The knowledge fragment entity corresponds to each document slice, and each document slice corresponds to one or more element entities. The knowledge fragment entity contains three attributes: representation vector, text information, and number of elements.

[0169] The relation types are as follows:

[0170] BELONGS_TO (belongs to): describes which category a document belongs to. For example, "Gas Management Regulations" BELONGS_TO "Management Regulations" indicates that "Gas Management Regulations" belongs to the Management Regulations category.

[0171] IN_DOMAIN (Domain): Describes which sub-domain of emergency safety a document type belongs to. For example, "Management Regulations" IN_DOMAIN "Gas" indicates that the Management Regulations belong to the gas field of emergency safety.

[0172] HAS_PAGE (Containing Page): Describes which pages a document includes. For example, "Report" HAS_PAGE Page 1 indicates that the "Report" is located on page 1.

[0173] LOCATED_IN (located on): Describes which page an element is located on. For example, Table 3 LOCATED_IN Page 5 indicates that Table 3 is located on page 5.

[0174] COMPRISED_OF (Composition): Describes the elements contained in a knowledge fragment. For example, knowledge fragment ACOMPRISED_OF Table 3 + Text Paragraph 5 indicates that the knowledge fragment contains Table 3 and Text Paragraph 5.

[0175] CITES (Citation): Describes which documents the current document references. For example, an academic paper CITES another document indicates that the academic paper references another document.

[0176] This knowledge graph can trace the source of each knowledge fragment retrieved to the specific document, page, and location of that fragment, facilitating easy querying and confirmation.

[0177] It should be noted that this multimodal embedding model can be trained on rich text documents, primarily PDFs, using YOLOv8 (a deep learning model architecture) as training data. This results in a rich text document element extraction model that extracts various elements from PDF documents, such as figures, tables, chapter titles, text paragraphs, figure titles, and figure annotations. Compared to the existing RAG system's retrieval module's inaccurate processing of figure information in rich text documents, the embodiments of this application can improve the accuracy of extracting domain knowledge from figures.

[0178] In summary, for example, Figure 10 A schematic diagram illustrating a retrieval enhancement method based on a multimodal interactive intelligent agent provided in an embodiment of this application, as shown below. Figure 10 As shown, the method includes:

[0179] 1. Obtain initial prompts, which may include images and vague user requests. For example, a vague user request could be "Are there any security risks this week?".

[0180] 2. Based on the MCP server, the multimodal fusion retrieval service can first determine whether an initial prompt requires a search. If a search is required, the retrieval model is invoked to output the search results; if no search is required, step 3 is executed.

[0181] 3. Based on the modality of the first data in the initial prompt, determine the target interaction model. If the modality of the first data is text-based, the vertical domain large language model is determined as the target interaction model; if the modality of the first data includes non-textual modalities, the vertical domain multimodal interaction model (i.e., ...) is determined. Figure 10The target interaction model is the vertical multimodal reasoning big model (in the context of the large-scale model). Users can interact with the vertical multimodal reasoning big model or the vertical large language model, that is, through direct multi-turn dialogue.

[0182] 4. During multi-turn dialogues, the vertical multimodal reasoning big model or the vertical language big model can call the multimodal fusion retrieval service for retrieval through function calling.

[0183] 5. Based on the modalities of the second data in the search results, determine the target interaction model. If the modality of the second data is text-based, the vertical domain large language model is determined as the target interaction model; if the modalities of the second data include non-textual modalities, the vertical domain multimodal interaction model (i.e., ...) is determined. Figure 10 The vertical multimodal reasoning big model in the middle is the target interaction model.

[0184] Taking the interaction between a user and a vertical domain multimodal reasoning big data model as an example, the user can enter a question based on the search results: "Which safety regulations are violated in the scenario shown in the figure, and what are the corresponding penalties?" The multimodal reasoning big data model can answer this question.

[0185] This application also provides a retrieval enhancement method based on a multimodal interactive intelligent agent. This method is applied to the retrieval enhancement system based on a multimodal interactive intelligent agent as described in any of the foregoing embodiments. This method is similar to the one described above. Figure 1 Similar to the above, it will not be repeated here.

[0186] The above are the method embodiments provided in this application. The apparatus provided in this application will be described below.

[0187] Figure 11 A schematic diagram of a retrieval enhancement device based on a multimodal interactive intelligent agent provided in this application is shown below. Figure 11 As shown, the retrieval enhancement device 400 based on multimodal interactive intelligent agents provided in this embodiment includes: an acquisition module 401, a retrieval module 402, and an interaction module 403.

[0188] Acquisition module 401 is used to acquire an initial prompt; the initial prompt includes first data of at least one modality.

[0189] The retrieval module 402 is used to perform a retrieval based on the initial prompts by a multimodal interactive intelligent agent when a retrieval operation is determined based on the initial prompts, and to obtain retrieval results, which include: second data of at least one modality.

[0190] The interaction module 403 is used to determine a target interaction model from multiple candidate interaction models through a multimodal interactive agent based on the modality of the first data and the modality of the second data; and to interact with the user through the target interaction model based on the initial prompts and search results.

[0191] The retrieval enhancement device based on multimodal interactive intelligent agents provided in this embodiment can execute the methods provided in any of the above method embodiments. The implementation principle and technical effect are similar, and will not be described in detail here.

[0192] Figure 12 This is a schematic diagram of the structure of an electronic device provided in this application. Figure 12 As shown, the electronic device 500 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the device 500 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0193] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0194] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0195] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0196] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0197] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0198] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0199] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0200] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0201] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0202] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0203] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0204] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0205] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0206] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0207] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A retrieval enhancement system based on a multimodal interactive intelligent agent, characterized in that, The retrieval enhancement system is used for: Obtain an initial prompt; the initial prompt includes first data of at least one modality; When a retrieval operation is determined based on the initial prompt by a multimodal interactive intelligent agent, a retrieval model is used to perform a retrieval based on the initial prompt to obtain retrieval results, which include: second data in at least one modality; The multimodal interactive agent determines a target interactive model from multiple candidate interactive models based on the modality of the first data and the modality of the second data; and interacts with the user based on the initial prompts and the search results using the target interactive model.

2. The retrieval enhancement system according to claim 1, characterized in that, The plurality of candidate interaction models includes: a vertical domain multimodal interaction model and a vertical domain large language model. The determination of the target interaction model from the plurality of candidate interaction models based on the modality of the first data and the modality of the second data includes: In response to the fact that the modality of the first data is text modality and the modality of the second data is text modality, the vertical domain large language model is determined to be the target interaction model; or, In response to the first data including a non-textual modality, and / or the second data including a non-textual modality, the vertical multimodal interaction model is determined as the target interaction model.

3. The retrieval enhancement system according to claim 2, characterized in that, The retrieval enhancement system is also used for: Through the multimodal interactive agent, in response to the initial prompt, it is determined that no retrieval operation will be performed, and the modality of the first data is text modality, and the vertical domain large language model is determined as the target interaction model; or, Through the multimodal interactive agent, in response to the initial prompt, it is determined that no retrieval operation will be performed, and the modality of the first data includes the non-text modality, thus determining the vertical domain multimodal interaction model as the target interaction model.

4. The retrieval enhancement system according to claim 2 or 3, characterized in that, The retrieval enhancement system is also used for: Receive the user-triggered interaction model switching instruction; the interaction model switching instruction is used to instruct the target interaction model to be switched to the vertical domain large language model; In response to the interaction model switching command, the multimodal interactive agent outputs a prompt message when the modality of the first data includes a non-text modality and / or the modality of the second data includes a non-text modality. The prompt message is used to instruct the vertical domain large language model not to reference the non-text modality data.

5. The retrieval enhancement system according to claim 2 or 3, characterized in that, The retrieval enhancement system is also used for: Obtain a vertical multimodal sample dataset, which includes: vertical multimodal sample cues and vertical sample responses; The preset model is trained using the vertical multimodal sample dataset to obtain the vertical multimodal interaction model. The preset model includes: a hidden layer, a text output module, and a multimodal feature vector output module; the hidden layer is used to extract semantic features based on the multimodal sample prompts to obtain semantic features that characterize the multimodal sample prompts; the text output module is used to output predicted text describing the multimodal sample prompts based on the semantic features; the multimodal feature vector output module is used to output predicted multimodal features characterizing the multimodal sample prompts based on the semantic features; the predicted text, the predicted multimodal features, and the vertical sample responses are used to train the preset model.

6. The retrieval enhancement system according to claim 5, characterized in that, The step of training a preset model using the vertical multimodal sample dataset to obtain the vertical multimodal interaction model includes: The trainable parameters of the multimodal feature vector output module are frozen, and a single-round dialogue training is performed on the part of the preset model other than the multimodal feature vector output module using the vertical multimodal sample dataset to obtain the first initial vertical multimodal interaction model. The trainable parameters of the multimodal feature vector output module are frozen, and the parts of the first initial vertical multimodal interaction model other than the multimodal feature vector output module are trained through multiple rounds of dialogue using the vertical multimodal sample dataset to obtain the second initial multimodal reasoning large model. The trainable parameters of the multimodal feature vector output module are frozen, and reinforcement learning training is performed on the part of the second initial multimodal inference large model other than the multimodal feature vector output module using the vertical multimodal sample preference dataset to obtain the multimodal inference large model.

7. The retrieval enhancement system according to any one of claims 1-3, characterized in that, The step of interacting with the user through the target interaction model, based on the initial prompts and the search results, includes: Through the target interaction model, based on the initial prompts and the search results, multiple rounds of interaction are conducted with the user; Any round of interaction in the multi-round interaction process includes at least the following steps: Step 1: Through the target interaction model, in response to the interaction information of this round of interaction based on the user input, determine to perform a retrieval operation, call the retrieval model, and perform a retrieval based at least on the interaction information to obtain new retrieval results; Step 2: Based on the new search results, a new target interaction model is determined from the multiple candidate interaction models; the new target interaction model is used to conduct the next round of interaction with the user based on the interaction information and the new search results.

8. A retrieval enhancement method based on a multimodal interactive intelligent agent, characterized in that, The method includes: Obtain an initial prompt; the initial prompt includes first data of at least one modality; When a retrieval operation is determined based on the initial prompt by a multimodal interactive intelligent agent, a retrieval model is used to perform a retrieval based on the initial prompt to obtain retrieval results, which include: second data in at least one modality; The multimodal interactive agent determines a target interactive model from multiple candidate interactive models based on the modality of the first data and the modality of the second data; and interacts with the user based on the initial prompts and the search results using the target interactive model.

9. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in claim 8.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of claim 8.

Images