Search enhancement system and method based on multi-modal interactive agents

Abstract

The application discloses a retrieval enhancement system and method based on a multi-modal interactive agent, and relates to the field of artificial intelligence.The retrieval enhancement system comprises the following steps: obtaining an initial prompt; the initial prompt comprises first data of at least one mode; through a multi-modal interactive agent, when it is determined that a retrieval operation is to be performed based on the initial prompt, a retrieval model is used to perform retrieval based on the initial prompt to obtain a retrieval result, the retrieval result comprises second data of at least one mode; through the multi-modal interactive agent, a target interactive model is determined from a plurality of candidate interactive models based on the mode of the first data and the mode of the second data; and the target interactive model is used to interact with a user based on the initial prompt and the retrieval result.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to a retrieval enhancement system and method based on a multi-modal interactive agent. BACKGROUND

[0002] A question answering system (Question Answering System) as an important application of information retrieval and natural language processing technology is usually composed of two core modules of retriever and reader. The system can accurately retrieve and generate natural language answers from massive data.

[0003] After the user obtains the retrieval result through the question answering system, the user needs to interact with the question answering system to obtain the final answer or further information. However, the existing technology usually adopts a fixed interaction model, which is difficult to meet the diversified needs of users for interaction mode and quality in different scenarios.

[0004] Therefore, how to improve the accuracy of interaction is a problem to be solved. SUMMARY

[0005] The embodiments of the present application provide a retrieval enhancement system and method based on a multi-modal interactive agent to improve the accuracy of interaction.

[0006] In a first aspect, the embodiments of the present application provide a retrieval enhancement system based on a multi-modal interactive agent, and the retrieval enhancement system is used to:

[0007] obtain an initial prompt; the initial prompt includes first data of at least one modality;

[0008] determine, by a multi-modal interactive agent, that a retrieval operation is performed based on the initial prompt, perform retrieval based on the initial prompt by a retrieval model, and obtain a retrieval result, the retrieval result including second data of at least one modality;

[0009] determine, by the multi-modal interactive agent, a target interaction model from a plurality of candidate interaction models based on the modality of the first data and the modality of the second data; and interact with the user based on the initial prompt and the retrieval result by the target interaction model.

[0010] In a possible implementation, the plurality of candidate interaction models includes a vertical multi-modal interaction model and a vertical large language model, and determining the target interaction model from the plurality of candidate interaction models based on the modality of the first data and the modality of the second data includes:

[0011] In response to the modality of the first data being a text modality and the modality of the second data being a text modality, determining that the vertical large language model is the target interaction model;

[0012] Alternatively,

[0013] In response to the modality of the first data including a non-text modality and / or the modality of the second data including a non-text modality, determining that the vertical multi-modal interaction model is the target interaction model.

[0014] In a possible implementation, the retrieval enhancement system is further configured to:

[0015] In response to determining, by the multi-modal interaction agent, based on the initial prompt, that no retrieval operation is to be performed and the modality of the first data is a text modality, determining that the vertical large language model is the target interaction model.

[0016] Alternatively,

[0017] In response to determining, by the multi-modal interaction agent, based on the initial prompt, that no retrieval operation is to be performed and the modality of the first data includes the non-text modality, determining that the vertical multi-modal interaction model is the target interaction model.

[0018] In a possible implementation, the retrieval enhancement system is further configured to:

[0019] receiving the user-triggered interaction model switching instruction; the interaction model switching instruction is used to instruct to switch the target interaction model to the vertical large language model;

[0020] In response to the interaction model switching instruction, outputting, by the multi-modal interaction agent, prompt information when the modality of the first data includes a non-text modality and / or the modality of the second data includes a non-text modality, the prompt information being used to indicate that the vertical large language model does not reference data of the non-text modality.

[0021] In a possible implementation, the retrieval enhancement system is further configured to:

[0022] obtaining a vertical multi-modal sample data set, the vertical multi-modal sample data set including: a vertical multi-modal sample prompt and a vertical sample response;

[0023] training, by using the vertical multi-modal sample data set, a preset model to obtain the vertical multi-modal interaction model;

[0024] The preset model comprises: a hidden layer, a text output module, and a multi-modal feature vector output module; the hidden layer is used for performing semantic feature extraction based on the multi-modal sample prompt to obtain semantic features for representing the multi-modal sample prompt; the text output module is used for outputting predicted text for describing the multi-modal sample prompt based on the semantic features; the multi-modal feature vector output module is used for outputting predicted multi-modal features for representing the multi-modal sample prompt based on the semantic features; and the predicted text, the predicted multi-modal features, and the vertical sample response are used for training the preset model.

[0025] In a possible implementation, the training of the preset model through the vertical multi-modal sample dataset to obtain the vertical multi-modal interaction model comprises:

[0026] The trainable parameters of the multi-modal feature vector output module are frozen, and the part of the preset model other than the multi-modal feature vector output module is trained through the vertical multi-modal sample dataset to obtain a first initial vertical multi-modal interaction model;

[0027] The trainable parameters of the multi-modal feature vector output module are frozen, and the part of the first initial vertical multi-modal interaction model other than the multi-modal feature vector output module is trained through the vertical multi-modal sample dataset to obtain a second initial multi-modal reasoning large model;

[0028] The trainable parameters of the multi-modal feature vector output module are frozen, and the part of the second initial multi-modal reasoning large model other than the multi-modal feature vector output module is trained through the vertical multi-modal sample preference dataset to obtain a multi-modal reasoning large model.

[0029] In a possible implementation, the interaction of the target interaction model with the user based on the initial prompt and the search result comprises:

[0030] The target interaction model is used to interact with the user based on the initial prompt and the search result in multiple rounds;

[0031] Any round of interaction in the multiple rounds of interaction at least comprises the following steps:

[0032] Step 1: The target interaction model is used to determine, in response to interaction information of the round of interaction based on the user input, that a search operation is to be performed, to call the search model to perform search based at least on the interaction information to obtain a new search result;

[0033] Step 2, based on the new search result, re-determine a new target interaction model from the plurality of candidate interaction models; the new target interaction model is used to perform the next round of interaction with the user based on the interaction information and the new search result.

[0034] In a second aspect, the embodiments of the present application provide a retrieval enhancement method based on a multi-modal interaction agent, the method comprising:

[0035] Obtaining an initial prompt; the initial prompt comprises first data of at least one modality;

[0036] Through the multi-modal interaction agent, when it is determined to perform a retrieval operation based on the initial prompt, the retrieval model is used to perform retrieval based on the initial prompt to obtain a retrieval result, the retrieval result comprising second data of at least one modality.

[0037] Through the multi-modal interaction agent, a target interaction model is determined from a plurality of candidate interaction models based on the modality of the first data and the modality of the second data; the target interaction model is used to interact with the user based on the initial prompt and the retrieval result.

[0038] In a third aspect, the embodiments of the present application provide a retrieval enhancement device based on a multi-modal interaction agent, comprising:

[0039] An obtaining module is configured to obtain an initial prompt; the initial prompt comprises first data of at least one modality;

[0040] A retrieval module is configured to, through the multi-modal interaction agent, when it is determined to perform a retrieval operation based on the initial prompt, use the retrieval model to perform retrieval based on the initial prompt to obtain a retrieval result, the retrieval result comprising second data of at least one modality.

[0041] An interaction module is configured to, through the multi-modal interaction agent, determine a target interaction model from a plurality of candidate interaction models based on the modality of the first data and the modality of the second data; the target interaction model is used to interact with the user based on the initial prompt and the retrieval result.

[0042] In a fourth aspect, the embodiments of the present application provide an electronic device, comprising a memory and a processor.

[0043] The memory stores computer execution instructions;

[0044] The processor executes the computer execution instructions stored in the memory, so that the processor executes the method of the second aspect.

[0045] In a fifth aspect, an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by a processor to implement the method in the second aspect.

[0046] In a sixth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the method in the second aspect.

[0047] The retrieval enhancement system and method based on the multi-modal interactive agent provided by the embodiments of the present application can first determine a retrieval operation based on an initial prompt, and obtain a retrieval result through a retrieval model. Secondly, a target interactive model is determined from a plurality of candidate interactive models in combination with the modal of the initial prompt and / or the retrieval result data, which can improve the pertinence and accuracy of the interaction. Moreover, the embodiments of the present application can expand the interaction with the user based on the initial prompt and / or the retrieval result, thereby enhancing the accuracy of the interaction. BRIEF DESCRIPTION OF DRAWINGS

[0048] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

[0050] Figure 1 A schematic diagram of a retrieval enhancement system based on a multi-modal interactive agent provided by an embodiment of the present application;

[0051] Figure 2 A flowchart of a method for obtaining an image-text pair provided by an embodiment of the present application;

[0052] Figure 3 A model architecture diagram of a preset model provided by an embodiment of the present application;

[0053] Figure 4 A flowchart of a method for training a preset model provided by an embodiment of the present application;

[0054] Figure 5 A schematic diagram of a training process of a multi-modal reasoning large model provided by an embodiment of the present application;

[0055] Figure 6 An architecture diagram of a retrieval model provided by an embodiment of the present application;

[0056] Figure 7 A schematic diagram of a training process of a multi-modal embedding model provided for an embodiment of the present application;

[0057] Figure 8 A schematic diagram of constructing a multi-modal database and a domain knowledge graph provided for an embodiment of the present application;

[0058] Figure 9 A schematic diagram of a process of constructing a domain knowledge graph provided for an embodiment of the present application;

[0059] Figure 10 A schematic diagram of a retrieval enhancement method based on a multi-modal interactive agent provided for an embodiment of the present application;

[0060] Figure 11 A structural schematic diagram of a retrieval enhancement device based on a multi-modal interactive agent provided for the present application;

[0061] Figure 12 A structural schematic diagram of an electronic device provided for the present application. DETAILED DESCRIPTION

[0062] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, the same numbers refer to the same or similar elements throughout the drawings. The implementation described in the following exemplary embodiments is not meant to represent all implementations consistent with the present application.

[0063] In the present application, the term “comprising” and its variants can refer to non-limiting inclusion; the term “or” and its variants can refer to “and / or”. In the present application, the terms “first”, “second”, etc. are used to distinguish similar objects, and do not necessarily mean a specific order or sequence. In the present application, “a plurality of” means two or more. “And / or”, which describes the association relationship between the associated objects, means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. The character “ / ” generally represents an “or” relationship between the front and rear associated objects.

[0064] A question answering system (Question Answering System) includes a retriever and a reader. The role of the retriever module is to retrieve relevant candidate fragments from large-scale corpus according to user questions, which contain the information needed to answer user questions. At present, the commonly used retriever includes sparse representation retrieval and dense vector retrieval. The reader module refines, reasons, and summarizes according to the relevant information retrieved, and finally gives a humanized (natural language form) answer.

[0065] Common readers use a Transformer (a deep learning model) architecture, which is generally divided into two types: information extraction type readers, such as BERT (a bidirectional encoder technique), RoBERTa (a pre-training method), etc., and text-to-text generation type readers, such as T5 (a text-to-text conversion transformer), BART (a bidirectional autoregressive transformer), GPT (a generative pre-training transformer), etc.

[0066] The role of the reader is to combine the search results to answer the user's question. Multi-modal large models have strong understanding and summarization capabilities due to their large number of parameters and training on massive amounts of data. With the development of large model technology and the improvement of application effects, readers have generally used multi-modal large models.

[0067] Searchers can be divided into several categories, such as same-modal search, cross-modal search, and multi-modal to multi-modal search. The search process generally uses an embedding (embedding vector) model to extract embedding vectors of all data to be searched. When performing a search, the query embedding vector is extracted using the same embedding model, and then compared with the embedding vectors of all data to be searched to obtain the top candidates with the highest similarity.

[0068] Currently, question and answer systems have developed to the Retrieval-Augmented Generation (RAG) stage, and continue to evolve into more powerful forms such as Advanced RAG, Modular RAG, Agentic RAG, and multi-modal RAG.

[0069] After obtaining the search results through the question and answer system, the user needs to interact with the question and answer system to obtain the final answer or further information. However, the existing technology generally uses a fixed interaction model, which is difficult to meet the diverse needs of users for interaction methods and quality in different scenarios.

[0070] For example, the fixed interaction model often limits the input method to a few common forms, such as only supporting pure text input. This single input method greatly limits the user's ability to express problems. In terms of output, the fixed interaction model usually uses standardized text formats to answer, and it is difficult to present through charts, videos, etc.

[0071] Therefore, the embodiment of the present application provides a retrieval enhancement system and method based on a multi-modal interactive agent, which can first determine a retrieval operation based on an initial prompt, and obtain a retrieval result through a retrieval model. Secondly, a target interactive model is determined from a plurality of candidate interactive models in combination with the modal of the initial prompt and / or the retrieval result data, which can improve the pertinence and accuracy of the interaction. Moreover, the embodiment of the present application can interact with the user based on the initial prompt and / or the retrieval result, thereby enhancing the accuracy of the interaction.

[0072] The technical solutions of the present application and how the technical solutions solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The embodiments of the present application will be described below with reference to the drawings.

[0073] Figure 1 A schematic diagram of a retrieval enhancement system based on a multi-modal interactive agent provided by the embodiment of the present application is shown in FIG. 1. As shown in the figure, the retrieval enhancement system comprises: Figure 1

[0074] S101, obtaining an initial prompt, the initial prompt comprising first data of at least one modal.

[0075] Optionally, the initial prompt can be the input content of the user, which can be a question, a demand description, etc. input by the user, and is the basis for subsequent retrieval and interaction. The modal can be the type of the first data, and different modals can represent different kinds of information. For example, the modal can be a text modal, an image modal, etc. any one or more.

[0076] For example, in the field of emergency safety, the initial prompt can be a textual description related to the field of emergency safety, and / or visual materials of the field of emergency safety. For example, an abnormal or emergency event segment of 10 seconds in length can be extracted at 1-3 frames per second, and a plurality of pictures of the abnormal or emergency event obtained thereby can constitute the initial prompt.

[0077] Optionally, the multi-modal interactive agent can be configured with an application programming interface (API), or a graphical user interface (GUI), etc. The multi-modal interactive agent can obtain the initial prompt input by the user through the API or GUI, etc.

[0078] ​S102, through the multi-modal interactive agent, based on the initial prompt, it is determined that the retrieval operation is performed, the retrieval model is used to retrieve based on the initial prompt, and retrieval results are obtained. The retrieval results include: at least one mode of second data.

[0079] Optionally, the retrieval model can have the ability to understand the meaning of the initial prompt, can convert the initial prompt into a computer processable query instruction, and search for relevant information according to a preset rule and algorithm to perform a retrieval operation. Consistent with the first data, the mode of the second data can also be any one or more of the text mode, the image mode, etc.

[0080] Optionally, the multi-modal interactive agent may, for example, include any existing large language model. The multi-modal interactive agent can construct a prompt word based on the above initial prompt (the prompt word can be used to indicate that the initial prompt is analyzed, and it is determined whether the initial prompt needs to be retrieved to respond to the initial prompt), and input the prompt word into the large language model to obtain a judgment result of whether the retrieval operation is needed.

[0081] For example, when the multi-modal interactive agent can directly answer the request and / or implicit request in the initial prompt based on its own database, the retrieval operation is not needed; for the request and / or implicit request in the initial prompt, the multi-modal interactive agent needs to call the retrieval model to answer, and then it is determined that the retrieval operation is needed.

[0082] S103, through the multi-modal interactive agent, based on the mode of the first data and the mode of the second data, a target interactive model is determined from a plurality of candidate interactive models; through the target interactive model, the initial prompt and the retrieval result are used to interact with the user.

[0083] Optionally, the plurality of candidate interactive models can each have an interactive logic, a processing manner and an applicable scene. For example, some candidate interactive models are good at processing the interaction between text and text, such as text-based question and answer interaction; some candidate interactive models are more suitable for processing the interaction scene of text and image combination, such as comprehensive analysis and reply according to the user input text description and uploaded image; some candidate interactive models can process multi-modal input and output, such as processing the input and output of text and output image at the same time.

[0084] The multi-modal interactive agent can determine the modality of the first data, for example, by reading the format identifier of the first data. The multi-modal interactive agent can determine the modality of the second data, for example, by reading the format identifier of the second data. Optionally, the multi-modal interactive agent can store a mapping relationship between the modality of the data and the candidate interaction model. The multi-modal interactive agent can determine the target interaction model from the plurality of candidate interaction models based on the modality of the first data, the modality of the second data, and the mapping relationship between the modality of the data and the candidate interaction model.

[0085] Optionally, the interaction with the user can be a process of information exchange and interaction between the user. The multi-modal interactive agent can receive an initial prompt input by the user through the target interaction model, analyze and process the initial prompt and the retrieval result by using the target interaction model, and then generate a corresponding reply to feed back to the user. For example, the target interaction model can refer to any existing large language model. The feedback of the user can trigger further interaction of the multi-modal interactive agent, forming an interaction cycle.

[0086] Optionally, the interaction process can be single-round interaction or multi-round interaction. Taking multi-round interaction as an example, any round of interaction in multi-round interaction at least includes the following steps:

[0087] Step 1, determining to perform a retrieval operation by using the target interaction model in response to the interaction information of the round of interaction based on the user input, calling a retrieval model, and performing retrieval based on at least the interaction information to obtain a new retrieval result.

[0088] Optionally, the interaction information is similar to the initial prompt, which can include data of at least one modality. The process of determining to perform a retrieval operation by the multi-modal interactive agent based on the interaction information and calling a retrieval model is similar to the above step S102, which will not be described here.

[0089] The retrieval enhancement system can perform retrieval based on the interaction information to obtain a new retrieval result. The retrieval enhancement system can also perform retrieval based on the interaction information of the round of interaction and the interaction information of the previous round to obtain a new retrieval result. The retrieval enhancement system can also perform retrieval based on the interaction information of the round of interaction, the interaction information of the previous round, and the initial prompt. The present application does not limit this.

[0090] Step 2, determining a new target interaction model from the plurality of candidate interaction models based on the new retrieval result, wherein the new target interaction model is used for the next round of interaction with the user based on the interaction information and the new retrieval result.

[0091] Optionally, the retrieval enhancement system can determine a new target interaction model from the plurality of candidate interaction models based on the modality of the new retrieval result.

[0092] The embodiment of the application realizes the function of dynamically adjusting the retrieval strategy and the interaction model according to the real-time input of the user through the multi-round interaction mechanism. Each round of interaction is optimized based on the result of the previous round, ensuring the continuity and pertinence of the interaction, and improving the speed of responding to the user's retrieval and the interaction quality.

[0093] The embodiment of the application can first determine the retrieval operation based on the initial prompt, and obtain the retrieval result through the retrieval model. Second, the target interaction model is determined from a plurality of candidate interaction models in combination with the modalities of the initial prompt and / or the retrieval result data, which can improve the pertinence and accuracy of the interaction. Moreover, the embodiment of the application can interact with the user based on the initial prompt and / or the retrieval result, thereby enhancing the accuracy of the interaction.

[0094] Taking the plurality of candidate interaction models including the vertical domain multi-modal interaction model and the vertical domain large language model as an example, the following will describe in detail how the multi-modal interaction agent determines the target interaction model from the plurality of candidate interaction models based on the modalities of the first data and the modalities of the second data.

[0095] In the case that the modality of the first data is a text modality and the modality of the second data is a text modality, the multi-modal interaction agent can determine the vertical domain large language model as the target interaction model.

[0096] Optionally, the vertical domain large language model can be a large language model capable of in-depth analysis and processing of text data in a vertical domain, and providing text interaction services. For example, the vertical domain large language model can be trained based on vertical domain data on the basis of a large language model. The vertical domain can be a specific vertical domain, for example, an emergency safety domain.

[0097] In the case that the modality of the first data includes a non-text modality, and / or the modality of the second data includes a non-text modality, the multi-modal interaction agent can determine the vertical domain multi-modal interaction model as the target interaction model.

[0098] Optionally, the vertical domain multi-modal interaction model can be a model capable of processing and integrating multiple different types of data modalities, such as text, image, audio, etc., in a vertical domain, and analyzing and interacting with input containing multiple modal data.

[0099] In an implementation manner, the multi-modal interaction agent can obtain a vertical domain multi-modal sample data set, which includes a vertical domain multi-modal sample prompt and a vertical domain sample response.

[0100] Optionally, the vertical domain multimodal sample prompt and the vertical domain sample response can be an image-text pair, wherein the image in the image-text pair can correspond to the vertical domain sample response, and the text in the image-text pair can correspond to the vertical domain multimodal sample prompt. The multimodal interactive agent can be configured with an API through which an open source database is accessed and the image-text pair is directly obtained from the open source database. Alternatively, the image-text pair can be obtained by processing the to-be-processed picture, table, or video.

[0101] Taking that the multimodal interactive agent can process the to-be-processed picture, table, or video to obtain the image-text pair as an example, Figure 2 A flowchart of a method for obtaining an image-text pair provided by an embodiment of the present application is shown in Figure 2 As shown in the figure, the method for obtaining an image-text pair is as follows:

[0102] 1. An existing multimodal large model is used to process all to-be-processed pictures, tables, and videos to obtain corresponding detailed text descriptions, i.e., to obtain an image-text pair.

[0103] 2. An existing another multimodal large model is used to score the text description of the image-text pair generated in step 1 for compliance, and the image-text pair with a text description compliance score higher than a threshold is retained. For example, the image-text pair with a text description compliance score higher than 8 is retained.

[0104] 3. An existing large language model is used to analyze the text description in all image-text pairs and filter out the image-text pairs that do not conform to the target domain.

[0105] 4. A visual encoder in an existing multimodal large model is used as a feature extractor for image data, and only the image data in all image-text pairs is extracted to obtain a feature vector, and the feature vectors of these images are stored in a vector database.

[0106] 5. The vectors in the vector database are filtered based on similarity. For example, only one of the images corresponding to the feature vectors with a similarity greater than 0.95 is retained, and a portion of the images with a similarity between 0.85 and 0.95 is randomly removed at a ratio of 50%, and finally a high-quality non-redundant image-text pair of the target domain is obtained.

[0107] After obtaining the vertical domain multimodal sample dataset, the retrieval enhancement system can also be used to train a preset model to obtain a vertical domain multimodal interactive model. The preset model can include a hidden layer, a text output module, and a multimodal feature vector output module. Figure 3 A model architecture diagram of a preset model provided by an embodiment of the present application is shown in Figure 3As shown, the preset model can include picture preprocessing, a visual encoder, which can be a Vision Transformer (a model architecture for image processing), a Pixel Unshuffle, a Projector, a Prompt, a Text Tokenizer, a Large Language Model (LLM), an LM Head, an Embedding Projector, an Output Text, and a Multi-vectors.

[0108] The preset model can also include a hidden layer, which is not shown in the figure. The hidden layer is used for semantic feature extraction based on the vertical domain multimodal sample prompt and the vertical domain sample response, to obtain semantic features for representing the vertical domain multimodal sample prompt and the vertical domain sample response. Figure 3

[0109] The text output module can be used to output predicted text based on the semantic features, for describing the vertical domain multimodal sample prompt and the vertical domain sample response.

[0110] Optionally, the text output module can deeply understand the association between the vertical domain multimodal sample prompt and the vertical domain sample response based on the semantic features, and then generate and output predicted text describing the vertical domain multimodal sample prompt and the vertical domain sample response. The predicted text can be used to train the preset model.

[0111] The multi-vectors output module is used to output predicted multi-modal features for representing the vertical domain multimodal sample prompt and the vertical domain sample response based on the semantic features.

[0112] Optionally, the multi-vectors output module can fuse the semantic features of the vertical domain multimodal sample prompt and the vertical domain sample response, extract key semantic elements therefrom, and obtain the predicted multi-modal features based on the mapping relationship between the key semantic elements and the multi-modal features. The predicted multi-modal features can be used to train the preset model.

[0113] By constructing a vertical domain multimodal sample dataset and training a preset model, the embodiments of the present application can obtain a vertical domain multimodal interaction model that can process text and multi-modal features simultaneously. The model has strong semantic understanding capability and can also output multi-modal feature vectors, providing a rich information basis for subsequent interactions and improving the accuracy and diversity of interactions.

[0114] Figure 4 A flowchart of a method for training a preset model according to an embodiment of the present application is shown in FIG. 6. Figure 4 ​As shown, the training process of the preset model can be as follows:

[0115] S201, freeze the trainable parameters of the multi-modal feature vector output module, and train the part of the preset model except the multi-modal feature vector output module through the vertical domain multi-modal sample data set for single-turn dialogue training to obtain a first initial vertical domain multi-modal interaction model.

[0116] Optionally, the trainable parameters can be parameters in the preset model that can be adjusted through optimization algorithms during training. Single-turn dialogue can be a one-time interaction between a user and a preset model, in which the user asks a question or gives an instruction, and the preset model generates a corresponding reply according to the input. Single-turn dialogue training is to let the preset model learn how to generate accurate, reasonable and contextually appropriate replies according to the input single-turn dialogue data.

[0117] The first initial vertical domain multi-modal interaction model can be a model obtained by training the part of the preset model except the multi-modal feature vector output module for single-turn dialogue training. The first initial vertical domain multi-modal interaction model is the product of preliminary training in the vertical domain multi-modal environment and has a certain vertical single-turn dialogue interaction capability, and can handle simple multi-modal interaction tasks within the vertical domain.

[0118] S202, freeze the trainable parameters of the multi-modal feature vector output module, and train the part of the first initial vertical domain multi-modal interaction model except the multi-modal feature vector output module through the vertical domain multi-modal sample data set for multi-turn dialogue training to obtain a second initial multi-modal reasoning large model.

[0119] Optionally, multi-turn dialogue can be multiple interactions between a user and the first initial vertical domain multi-modal interaction model. After the user asks a question or gives an instruction in the previous turn and the first initial vertical domain multi-modal interaction model gives a reply, the user may further ask questions or provide more information based on the reply, and the first initial vertical domain multi-modal interaction model needs to combine context information for subsequent replies. Multi-turn dialogue training is to let the model learn how to accurately understand user intent and generate coherent and reasonable replies in such a dialogue containing multiple interaction rounds.

[0120] The second initial multi-modal reasoning large model has stronger ability in handling multi-turn dialogue and reasoning than the first initial model, and can better adapt to complex multi-turn interaction scenarios within the vertical domain, better understand user needs and give reasonable reasoning results.

[0121] S203, freeze the trainable parameters of the multi-modal feature vector output module, and perform reinforcement learning training on the part of the second initial multi-modal reasoning large model except the multi-modal feature vector output module through the vertical domain multi-modal sample preference data set, to obtain a multi-modal reasoning large model.

[0122] Optionally, the vertical domain multi-modal sample preference data set can be a sample set of multi-modal data and corresponding preference information. Reinforcement learning allows an agent to learn in interaction with an environment, optimizing its behavior strategy through continuous trial and feedback. During the reinforcement learning training process, the agent selects an action according to the current state, and the environment gives a reward or punishment signal according to the action, and the agent adjusts its strategy according to the signal to maximize the long-term cumulative reward.

[0123] The multi-modal reasoning large model can be a model obtained by performing reinforcement learning training on the second initial multi-modal reasoning large model. It has powerful multi-modal data processing and reasoning ability in the vertical domain, and can integrate information of multiple modalities, give accurate, reasonable and user-expected reasoning results according to the user's input and preferences.

[0124] Exemplarily, Figure 5 A schematic diagram of a training process of a multi-modal reasoning large model provided for the application embodiment is shown in Figure 5 The training process of the multi-modal reasoning large model is as follows:

[0125] 1. Freeze the Embedding task, and only train the LM Head (language model head) through single round dialogue on the vertical domain multi-modal data set.

[0126] 2. Freeze the Embedding task, and only train the LM Head through multi-round dialogue on the vertical domain multi-modal data set.

[0127] 3. Freeze the Embedding task, and only train the LM Head through reinforcement learning on the vertical domain multi-modal data set (i.e. vertical domain multi-modal sample preference data set).

[0128] Optionally, the vertical domain multi-modal sample preference data set is a reconstruction of the vertical domain multi-modal sample data set, and the reconstruction method can refer to the existing implementation method.

[0129] The application embodiment can gradually optimize the performance of the vertical domain multi-modal interaction model through the phased training strategy. The phased training strategy can first perform single round dialogue training to enable the model to have basic interaction ability, then perform multi-round dialogue training to enhance the context understanding and continuous interaction ability of the model, and finally perform reinforcement learning training to further improve the reasoning and decision-making ability of the model, thereby effectively improving the interaction quality and stability of the model.

[0130] The embodiment of the application selects a suitable interaction model based on the modal type of the first data and the second data. When only text is involved, a vertical large language model is selected for efficient processing; when non-text modalities are involved, a vertical multi-modal interaction model is switched to, which can ensure comprehensive understanding and accurate response of multi-modal information and improve the accuracy of interaction.

[0131] The above embodiment illustrates that the multi-modal interaction agent determines the relevant content for which the retrieval operation is performed in response to the initial prompt. In the following, the multi-modal interaction agent determining the relevant content for which the retrieval operation is not performed in response to the initial prompt will be introduced.

[0132] The retrieval enhancement system is also used to determine, by the multi-modal interaction agent, that the retrieval operation is not performed in response to the initial prompt. When the modal of the first data is a text modality, a vertical large language model can be determined as the target interaction model. When the modal of the first data includes a non-text modality, a vertical multi-modal interaction model can be determined as the target interaction model.

[0133] Optionally, the multi-modal interaction agent may, for example, include any existing large language model. The multi-modal interaction agent can construct a prompt word (which can be used to indicate that the initial prompt is analyzed to determine whether retrieval is required in response to the initial prompt) based on the above initial prompt, and input the prompt word into the large language model to obtain a judgment result of whether retrieval is required. When the judgment result indicates that retrieval is not required, the multi-modal interaction agent can determine that the retrieval operation is not performed.

[0134] For example, when the user input contains a picture or the retrieval result contains a picture, a vertical multi-modal interaction model is selected to answer the user's question; when the user input does not contain a picture or the retrieval result does not contain a picture, a vertical large language model is selected to answer the user's question.

[0135] Optionally, the vertical large language model and the vertical multi-modal interaction model are similar to the above, which will not be described here.

[0136] The embodiment of the application provides a model selection strategy when there is no retrieval requirement, which can determine the interaction model according to the modal type of the initial prompt. For pure text input, a vertical large language model is directly used for quick response; for input containing non-text modalities, a vertical multi-modal interaction model is enabled to ensure accurate interaction in different scenarios.

[0137] After the multi-modal interaction agent determines the target interaction model, the multi-modal interaction agent can be configured with a GUI to receive a user-triggered interaction model switching instruction, which is used to indicate that the target interaction model is switched to a vertical large language model.

[0138] Optionally, the user-triggered interaction model switching instruction can be triggered through an interface for interaction between the multi-modal interaction agent and the user. For example, the switching instruction can be any one or more of a voice instruction, a text input, a gesture operation, etc.

[0139] The multi-modal interaction agent can be configured with a GUI. In response to the interaction model switching instruction, when the modality of the first data includes a non-text modality and / or the modality of the second data includes a non-text modality, the multi-modal interaction agent outputs prompt information through the GUI. The prompt information is used to indicate that the vertical large language model does not reference data of the non-text modality.

[0140] Optionally, the multi-modal interaction agent can be configured with a GUI. In response to the interaction model switching instruction, when the modality of the first data includes a non-text modality, the multi-modal interaction agent outputs prompt information through the GUI. The multi-modal interaction agent can also output prompt information in response to the interaction model switching instruction when the modality of the second data includes a non-text modality. The multi-modal interaction agent can also output prompt information in response to the interaction model switching instruction when the modality of the first data includes a non-text modality and the modality of the second data includes a non-text modality.

[0141] Optionally, the multi-modal interaction agent can output prompt information in the form of a pop-up window. For example, the prompt information can be "Please note that the vertical large language model cannot reference pictures and video content in the conversation history". The vertical large language model can use a placeholder marker to occupy the data of the non-text modality and then continue to answer.

[0142] The embodiments of the present application introduce a model switching instruction and corresponding prompt information. When the user attempts to switch the interaction model to a vertical large language model that is not suitable for the current non-text modality data, the system gives an explicit prompt, avoiding invalid or incorrect interactions and improving the accuracy of the interaction.

[0143] On the basis of the above-mentioned embodiments, Figure 6 An architecture schematic diagram of a retrieval model provided by the embodiments of the present application is shown in Figure 6 As shown, the retrieval model can respond to user input text and / or input pictures. The input text is rewritten into multiple queries, for example, query1 and query2, through a Query (query) rewriting and intent recognition large model.

[0144] For example, the input text of the user can be "check if the current monitoring system captures the abnormal behavior of the staff such as fighting, smoking, falling, etc.", and the output of the rewritten query model is: query1: staff smoking, query2: staff fighting, query3: climbing over the wall.

[0145] The rewritten input text and / or the input picture are input into a multi-modal embedding model. The architecture of the multi-modal embedding model can refer to the above Figure 3 , which can include a text tokenizer and a ViT (a model architecture for image processing). Figure 7 The schematic diagram of the training process of the multi-modal embedding model provided by the embodiments of the present application is shown in Figure 7 , and the training process of the multi-modal embedding model can be trained by the following steps:

[0146] 1. Freeze the Embedding task and only train the LM Head (Language Model Head) on the vertical multi-modal dataset. It should be noted that the vertical multi-modal dataset can be a vertical multi-modal sample dataset.

[0147] 2. Freeze the text generation task and only train the Embedding task on the vertical multi-modal dataset.

[0148] 3. For the same batch of training data, randomly start one of the generation or Embedding tasks with a probability of 50%, and perform both tasks with a probability of 50%. The training data can include a vertical multi-modal dataset and an optical character recognition (OCR) dataset. The vertical multi-modal dataset can account for 70%, and the OCR dataset can account for 30%.

[0149] 4. All batches of training data are performed simultaneously, and only one batch (epoch) is executed in this stage. The training data is consistent with the training data in step 3.

[0150] The multi-modal embedding model can provide cross-modal retrieval services based on the Model Context Protocol (MCP), including basic semantic retrieval, cross-modal semantic retrieval, sparse retrieval / keyword, and hybrid retrieval. The cross-modal retrieval service can be based on a multi-modal database and a domain knowledge graph for retrieval, and finally obtain the retrieval result.

[0151] Current multi-modal RAG has two solutions to handle multi-modal problems. One is to convert other modal inputs into text first, and then use a large language model for retrieval and enhancement generation. The other is to use a multi-modal embedding model to retrieve the text and picture input by the user, and then use a multi-modal large model for multi-turn dialogue.

[0152] The first method has two disadvantages. First, the conversion of other modalities to text modalities relies on a high degree of information loss, which can result in less than ideal retrieval and enhancement results. Second, the conversion of other modalities to text modalities requires the use of a large multi-modal model, which has a large computational load. The second method avoids information loss, but traditional multi-modal embedding models can only convert image or text information into embedding vectors. If sparse retrieval or mixed retrieval is to be performed on other modalities other than text, it cannot be achieved.

[0153] Therefore, the retrieval model provided by the embodiments of the present application can implement cross-modal retrieval (such as text-to-image, image-to-text, or mixed retrieval) compared to the multi-modal RAG in the prior art, cover complex query scenarios, and achieve multi-modal retrieval.

[0154] The construction of the multi-modal database and the field knowledge graph is described in detail below. Figure 8 A schematic diagram of constructing a multi-modal database and a field knowledge graph according to an embodiment of the present application is shown in Figure 8 As shown, images and videos can obtain a multi-modal database based on a multi-modal embedding model and system prompts. The multi-modal database can include Multi-Vectors (multi-modal feature vectors) and text descriptions. Image-text documents and text documents can construct a field knowledge graph based on a rich text document parsing module and a multi-modal embedding model.

[0155] The rich text document can be a document type that carries rich formats and elements such as font styles, pictures, tables, charts, audio and video, hyperlinks, in addition to text content. For the rich text document parsing module, Figure 9 A flowchart of constructing a field knowledge graph according to an embodiment of the present application is shown in Figure 9 As shown, image-text documents and text documents can include Portable Document Format (PDF), office (a document format), and plain text documents.

[0156] The multi-modal interactive agent can be configured with a conversion tool for converting documents into PDF format, by which the above domain knowledge documents can be all converted into PDF format, and then into images in Joint Photographic Experts Group (JPG) format. The layout element extraction model can perform layout element recognition on the above images, extract layout element numbers, and perform region segmentation based on pictures, tables, texts, titles, etc.

[0157] Optionally, the layout element extraction model can automatically identify and extract various layout elements from the images converted from the rich text documents. The layout element extraction model can use an existing deep learning model. For example, the layout elements can include any one or more of figures, tables, chapter names, text paragraphs, chart titles, chart annotations, etc. The attribute information of the layout elements can be information describing the characteristics and properties of the layout elements, for example, the attribute information of the layout elements can include layout element numbers.

[0158] Optionally, the layout element extraction model can first perform multiple batch processing on all images converted from the rich text documents until all images converted from the rich text documents are processed. Second, the layout element extraction model can cut out the regions of all elements in all pages according to the recognition results of each page to obtain pictures corresponding to all elements in all pages of all rich text documents. Third, the layout element extraction model can divide all element pictures into two categories corresponding to different system prompt words, namely text elements and chart elements. Finally, the layout element extraction model can group each category of element pictures by size to reduce the size difference of pictures in the same batch and avoid invalid calculation caused by alignment.

[0159] The multi-modal large model (i.e., the multi-modal embedding model) can extract element text and knowledge by batch processing of text elements and batch processing of figures and tables. The multi-modal large model (i.e., the multi-modal embedding model) can extract embedding vectors from sliced text, including recombining the recognition results of each element into a Markdown (a markup language) pure text document and extracting vectors based on document slicing. Optionally, the multi-modal interactive agent can include a document generation and writing module, by which the text information corresponding to all elements is written into a document in the order from top to bottom and from left to right to obtain a text document corresponding to the original rich text document.

[0160] In summary, the multi-modal interactive agent can construct a domain knowledge graph based on layout element numbers, document slicing extraction vectors, and document metadata, etc. by using any existing knowledge graph construction logic.

[0161] In an embodiment, the domain knowledge graph can contain six entity types and six relationship types, the entity types are as follows:

[0162] Domain (emergency safety subfield): gas, water, petrochemical, transportation, etc.

[0163] Doc Category (document type): document type 1, document type 2, document type 3, etc.

[0164] Document (document): paper, regulations, management regulations, etc.

[0165] Page (page): page1, page2, page3, etc.

[0166] Content Element (element): picture element, table element, title element, etc.

[0167] Knowledge Chunk (knowledge piece): document slice 1, document slice 2, document slice 3, etc.

[0168] Among them, the attributes contained in the page entity include four attributes: original document identifier (Identifier, ID), the number of pictures contained, the number of tables contained, and page overview; The element entity includes five attributes: original document ID, original page ID, save path, text information, and representation vector; The knowledge piece entity corresponds to each document slice, each document slice corresponds to one to multiple element entities, and the knowledge piece entity contains three attributes: representation vector, text information, and the number of contained elements.

[0169] The relationship types are as follows:

[0170] BELONGS_TO (belongs to): describes which category a document belongs to, for example, “Gas Management Regulations” BELONGS_TO “management regulations”, indicating that “Gas Management Regulations” belongs to the management regulations category.

[0171] IN_DOMAIN (domain): describes which subfield of emergency safety a document type belongs to, for example, “management regulations” IN_DOMAIN “gas”, indicating that management regulations belong to the gas field of emergency safety.

[0172] HAS_PAGE (contains page): describes which page a document includes, for example, “report” HAS_PAGE page1 (Page 1), indicating that “report” is located in page1.

[0173] LOCATED_IN (Located in): describes which page an element is located in, for example, Table 3 LOCATED_IN Page 5 (Page 5), which represents that Table 3 is located in Page 5.

[0174] COMPRISED_OF (Comprised of): describes which elements a certain piece of knowledge contains, for example, Knowledge Piece A COMPRISED_OF Table 3 + Text Passage 5, which represents that the knowledge piece contains Table 3 and Text Passage 5.

[0175] CITES (Cites): describes which document the current document refers to, for example, an academic paper CITES another literature, which represents that the academic paper refers to another literature.

[0176] The field knowledge graph can trace back to which page and which location in which document the knowledge piece comes from according to the knowledge piece obtained each time, facilitating query confirmation.

[0177] It should be noted that the multi-modal embedding model can be based on YOLOv8 (a deep learning model architecture) to obtain a rich text document element extraction model for extracting various elements in a PDF-based rich text document as training data, to extract elements such as figures, tables, chapter names, text passages, figure / table titles, and figure / table annotations in the rich text document. Compared with the problem that the retrieval module in the existing RAG system does not accurately process the figure / table information in the rich text document, the embodiment of the present application can improve the accuracy of extracting domain knowledge from figures and tables.

[0178] In summary, an exemplary, Figure 10 A schematic diagram of a retrieval enhancement method based on a multi-modal interactive agent provided by the embodiment of the present application is shown in Figure 10 As shown in the figure, the method comprises the following steps:

[0179] 1. Obtain an initial prompt, which can include a picture and a user's vague request. Exemplarily, the user's vague request can be "Is there any security risk this week?".

[0180] 2. Based on the MCP service (MCP server), the multi-modal fusion retrieval service can first determine whether the initial prompt needs to be retrieved. If retrieval is needed, the retrieval model is called to output the retrieval result; if retrieval is not needed, step 3 is performed.

[0181] 3. Determine the target interactive model based on the modality of the first data in the initial prompt. If the modality of the first data is text, determine the vertical domain large language model as the target interactive model; if the modality of the first data includes non-text, determine the vertical domain multi-modal interactive model (i.e. Figure 10The target interaction model is the vertical multimodal reasoning big model (in the context of the large-scale model). Users can interact with the vertical multimodal reasoning big model or the vertical large language model, that is, through direct multi-turn dialogue.

[0182] 4. During multi-turn dialogues, the vertical multimodal reasoning big model or the vertical language big model can call the multimodal fusion retrieval service for retrieval through function calling.

[0183] 5. Based on the modalities of the second data in the search results, determine the target interaction model. If the modality of the second data is text-based, the vertical domain large language model is determined as the target interaction model; if the modalities of the second data include non-textual modalities, the vertical domain multimodal interaction model (i.e., ...) is determined. Figure 10 The vertical multimodal reasoning big model in the middle is the target interaction model.

[0184] Taking the interaction between a user and a vertical domain multimodal reasoning big data model as an example, the user can enter a question based on the search results: "Which safety regulations are violated in the scenario shown in the figure, and what are the corresponding penalties?" The multimodal reasoning big data model can answer this question.

[0185] This application also provides a retrieval enhancement method based on a multimodal interactive intelligent agent. This method is applied to the retrieval enhancement system based on a multimodal interactive intelligent agent as described in any of the foregoing embodiments. This method is similar to the one described above. Figure 1 Similar to the above, it will not be repeated here.

[0186] The above are the method embodiments provided in this application. The apparatus provided in this application will be described below.

[0187] Figure 11 A schematic diagram of a retrieval enhancement device based on a multimodal interactive intelligent agent provided in this application is shown below. Figure 11 As shown, the retrieval enhancement device 400 based on multimodal interactive intelligent agents provided in this embodiment includes: an acquisition module 401, a retrieval module 402, and an interaction module 403.

[0188] Acquisition module 401 is used to acquire an initial prompt; the initial prompt includes first data of at least one modality.

[0189] The retrieval module 402 is used to perform a retrieval based on the initial prompts by a multimodal interactive intelligent agent when a retrieval operation is determined based on the initial prompts, and to obtain retrieval results, which include: second data of at least one modality.

[0190] The interaction module 403 is configured to determine, by the multi-modal interaction agent, the target interaction model from the plurality of candidate interaction models based on the modal of the first data and the modal of the second data, and interact with the user based on the initial prompt and the search result by using the target interaction model.

[0191] The retrieval enhancement apparatus based on the multi-modal interaction agent provided in the embodiment can execute the method provided in any of the method embodiments, and has similar implementation principles and technical effects. Details are not described herein again.

[0192] Figure 12 A structural schematic diagram of an electronic device is provided in the present application. As shown in the figure, Figure 12 The electronic device 500 provided in the embodiment includes at least one processor 501 and a memory 502. Optionally, the device 500 further includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected through a bus 504.

[0193] In the implementation process, the at least one processor 501 executes the computer execution instructions stored in the memory 502, so that the at least one processor 501 executes the method described above.

[0194] The specific implementation process of the processor 501 can refer to the method embodiments described above, and has similar implementation principles and technical effects. Details are not described herein again.

[0195] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as execution completed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0196] The memory can include a random access memory (RAM), and can also include a non-volatile memory (NVM), for example, at least one disk memory.

[0197] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0198] The present application also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described above.

[0199] The present application also provides a computer readable storage medium, which stores computer execution instructions, and when a processor executes the computer execution instructions, the method described above is implemented.

[0200] The readable storage medium described above can be implemented by any type of volatile or non-volatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0201] An exemplary readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium, and can write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0202] The division of units is only a logical functional division, and in actual implementation, there can be another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0203] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0204] In addition, each functional unit in various embodiments of the application can be integrated into one processing unit, or each unit can exist physically, or two or more units can be integrated into one unit.

[0205] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the application essentially or the part of the prior art that contributes to the technical solutions or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the method of the application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0206] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The aforementioned program can be stored in a computer readable storage medium. The program executes the steps including the above-mentioned method embodiments when executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, and various media that can store program codes.

[0207] Finally, it should be noted that those skilled in the art, after considering the specification and practicing the application disclosed herein, will easily think of other embodiments of the application. The application is intended to cover any variations, uses or adaptations of the application that follow the general principles of the application and include common knowledge or conventional technical means in the art that are not disclosed in the application, and is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof.

Claims

1. A multi-modal interaction agent based retrieval augmentation system, comprising: The retrieval enhancement system is used for: obtaining an initial prompt; the initial prompt includes first data of at least one modality; determining, by a multi-modal interactive agent, based on the initial prompt, that retrieval is to be performed, retrieving, by a retrieval model, based on the initial prompt, to obtain retrieval results, the retrieval results including second data of at least one modality; determining, by the multi-modal interactive agent, that a vertical domain large language model is a target interactive model in response to the modality of the first data being a text modality and the modality of the second data being a text modality; or, determining, by the multi-modal interactive agent, that a vertical domain multi-modal interactive model is a target interactive model in response to the modality of the first data including a non-text modality and / or the modality of the second data including a non-text modality; or, determining, by the multi-modal interactive agent, that the vertical domain large language model is the target interactive model in response to determining, based on the initial prompt, that retrieval is not to be performed and the modality of the first data being a text modality; or, determining, by the multi-modal interactive agent, that the vertical domain multi-modal interactive model is the target interactive model in response to determining, based on the initial prompt, that retrieval is not to be performed and the modality of the first data including the non-text modality; interacting, by the target interactive model, with a user based on the initial prompt and the retrieval results; The retrieval enhancement system is also used for: receiving an interactive model switching instruction triggered by the user; the interactive model switching instruction is used to instruct switching the target interactive model to the vertical domain large language model; determining, by the multi-modal interactive agent, in response to the interactive model switching instruction, that prompt information is to be output when the modality of the first data includes a non-text modality and / or the modality of the second data includes a non-text modality, the prompt information being used to indicate that the vertical domain large language model does not reference data of the non-text modality.

2. The retrieval enhancement system of claim 1, wherein, The retrieval enhancement system is also used for: obtaining a vertical domain multi-modal sample data set, the vertical domain multi-modal sample data set including a vertical domain multi-modal sample prompt and a vertical domain sample response; training a preset model based on the vertical domain multi-modal sample data set to obtain the vertical domain multi-modal interactive model; The preset model includes a hidden layer, a text output module, and a multi-modal feature vector output module; the hidden layer is used to perform semantic feature extraction based on the multi-modal sample prompt to obtain semantic features for representing the multi-modal sample prompt; the text output module is used to output predicted text for describing the multi-modal sample prompt based on the semantic features; the multi-modal feature vector output module is used to output predicted multi-modal features for representing the multi-modal sample prompt based on the semantic features; the predicted text, the predicted multi-modal features, and the vertical domain sample response are used to train the preset model.

3. The retrieval enhancement system of claim 2, wherein, The training of the preset model based on the vertical domain multi-modal sample data set to obtain the vertical domain multi-modal interactive model includes: freeze the trainable parameters of the multi-modal feature vector output module, and perform single-turn dialogue training on the part of the preset model other than the multi-modal feature vector output module through the vertical domain multi-modal sample data set, to obtain a first initial vertical domain multi-modal interaction model; freeze the trainable parameters of the multi-modal feature vector output module, and perform multi-turn dialogue training on the part of the first initial vertical domain multi-modal interaction model other than the multi-modal feature vector output module through the vertical domain multi-modal sample data set, to obtain a second initial multi-modal reasoning large model; freeze the trainable parameters of the multi-modal feature vector output module, and perform reinforcement learning training on the part of the second initial multi-modal reasoning large model other than the multi-modal feature vector output module through the vertical domain multi-modal sample preference data set, to obtain a multi-modal reasoning large model.

4. The retrieval enhancement system of claim 1, wherein, The interaction with the user based on the initial prompt and the retrieval result through the target interaction model includes: The target interaction model is used to interact with the user based on the initial prompt and the retrieval result through multiple turns; Any turn of the multi-turn interaction process at least includes the following steps: Step 1, the target interaction model is used to determine to perform a retrieval operation in response to interaction information of the turn of interaction based on the user input, call the retrieval model, and perform retrieval based on at least the interaction information to obtain a new retrieval result; Step 2, based on the new retrieval result, a new target interaction model is re-determined from a plurality of candidate interaction models; the new target interaction model is used to perform the next turn of interaction with the user based on the interaction information and the new retrieval result.

5. A method for retrieval augmentation based on multi-modal interaction agents, the method comprising: The method includes: An initial prompt is obtained; the initial prompt includes first data of at least one modality; When it is determined to perform a retrieval operation based on the initial prompt, the multi-modal interaction agent is used to perform retrieval based on the initial prompt through a retrieval model to obtain a retrieval result, and the retrieval result includes second data of at least one modality; In response to the modality of the first data being a text modality and the modality of the second data being a text modality, the vertical domain large language model is determined as the target interaction model through the multi-modal interaction agent; Or, In response to the modality of the first data including a non-text modality and / or the modality of the second data including a non-text modality, the vertical domain multi-modal interaction model is determined as the target interaction model through the multi-modal interaction agent; Or, In response to determining not to perform a retrieval operation based on the initial prompt and the modality of the first data being a text modality, the vertical domain large language model is determined as the target interaction model through the multi-modal interaction agent; Or, In response to determining not to perform a retrieval operation based on the initial prompt and the modality of the first data including the non-text modality, the vertical domain multi-modal interaction model is determined as the target interaction model through the multi-modal interaction agent; The target interaction model is used to interact with the user based on the initial prompt and the retrieval result; The retrieval enhancement system is also used for: receiving the user-triggered interaction model switching instruction; the interaction model switching instruction is used to instruct to switch the target interaction model to the vertical large language model; in response to the interaction model switching instruction, the multi-modal interaction intelligent agent outputs prompt information when the modality of the first data includes a non-text modality and / or the modality of the second data includes a non-text modality, the prompt information being used to instruct the vertical large language model not to reference data of the non-text modality.

6. An electronic device, comprising: comprising: a memory, a processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, so that the processor executes the method of claim 5.

7. A computer program product, characterised in that, comprising a computer program, which, when executed by a processor, implements the method of claim 5. comprising a computer program, which, when executed by a processor, implements the method of claim 5.

Images