User question response method and system, computer equipment and readable storage medium

Abstract

The invention relates to a user question response method and system, computer equipment and a readable storage medium. The method comprises the following steps: calling a large language model in response to a question of a user; in the process of outputting the large language model, when the large language model outputs one text unit, retrieval is carried out in other modal databases based on the text unit; the other modal database is constructed based on other modal data in the multi-modal data; the modals of the other modal data do not comprise texts; the output result of the large language model comprises at least one text unit; if the target other modal data is retrieved, assembling the text unit and the target other modal data into a reply of the question asked by the user; and if the target other modal data is not retrieved, taking an output result of the large language model as a reply of the question asked by the user. By adopting the method, the reply accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a user question response method, system, computer device and readable storage medium. Background Art

[0002] LLM (Large Language Model) is an artificial intelligence technology based on deep learning.

[0003] With the development of this technology, MLLM (Multimodal Large Language Models) have emerged, which can output multimodal data based on the questions entered by users, making it easier for users to understand the answers to the questions output by the large language model. Multimodal data refers to a data set containing two or more different types (i.e., modal) of information, such as a data set that includes at least two of the following: images, text, audio, and video. Modality refers to the form or type of data.

[0004] However, in traditional technologies, the responses output by multimodal large language models are often unsatisfactory. For example, the responses contain text data and other modal data, but the correlation between the text data and other modal data is low. Among them, other modal data refers to data other than text data in multimodal data, such as image data and video data. Summary of the invention

[0005] Based on this, it is necessary to provide a user question response method, system, computer device and readable storage medium that can improve the accuracy of answers to the above technical problems.

[0006] In a first aspect, the present application provides a method for responding to a user question, comprising: In response to user questions, a large language model is called; During the output of the large language model, each time the large language model outputs a text unit, a search is performed in other modality databases based on the text unit; the other modality databases are constructed based on other modality data in the multimodal data; the modality of the other modality data does not include text; the output result of the large language model includes at least one text unit; If the target other modal data is retrieved, assembling the text unit and the target other modal data into a reply to the user's question; If the target other modality data is not retrieved, the output result of the large language model is used as the answer to the user's question.

[0007] In one embodiment, in response to a user's question, calling a large language model includes: In response to a user's question, searching a text database based on the question carried in the user's question; the text database is constructed based on text data; If the first text data is retrieved, the question and the first text data are assembled as input of a large language model, and the large language model is called.

[0008] In one embodiment, after searching in other modality databases based on the text unit, the method includes: If at least one first other modal data is retrieved, data matching is performed on the first other modal data based on the mapping relationship between the text data and the other modal data and the first text data; If the match is successful, it is determined that the target other modal data is retrieved.

[0009] In one embodiment, searching in other modality databases based on the text unit includes: Based on the mapping relationship between the text data and the other modal data, determining second other modal data corresponding to the first text data for subsequent retrieval from the other modal database; A search is performed in the second other modality data based on the text unit.

[0010] In one embodiment, the method further comprises: Acquire multimodal data; In a case where the modality of the multimodal data includes text, recording a first mapping relationship between second text data and third other modality data in the multimodal data; The text database is obtained based on the second text data, and the other modality database is obtained based on the third other modality data.

[0011] In one embodiment, after acquiring the multimodal data, the method further includes: The multimodal data is fourth other modal data, and the third text data is extracted based on the fourth other modal data; Recording a second mapping relationship between the third text data and the fourth other modality data; The text database is obtained based on the third text data, and the other modality database is obtained based on the fourth other modality data.

[0012] In one embodiment, searching in other modality databases based on the text unit includes: Directly search in other modality databases based on the text unit; If the third other modal data is retrieved, the third other modal data is determined to be the target other modal data.

[0013] In one embodiment, if the target other modal data is retrieved, assembling the text unit and the target other modal data into a response to the user's question includes: Each time a target other modal data is retrieved, the target other modal data is placed between the current text unit and the next text unit until the large language model stops outputting, and an answer to the user's question is obtained.

[0014] In a second aspect, the present application also provides a user question response device, comprising: A calling module, used to call a large language model in response to a user's question; A retrieval module, configured to, during the output process of the large language model, search in other modality databases based on each text unit output by the large language model; the other modality databases are constructed based on other modality data in the multimodal data; the modality of the other modality data does not include text; and the output result of the large language model includes at least one text unit; An assembling module is used to assemble the text unit and the target other modal data into a reply to the user's question if the target other modal data is retrieved; if the target other modal data is not retrieved, use the output result of the large language model as the reply to the user's question.

[0015] In a third aspect, the present application also provides a user question response system, comprising: A collection interface, wherein an input box is provided in the collection interface for the user to input questions; The answer output interface is used to output the answer to the user's question assembled by the text unit and other target modal data; Wherein, the text unit is output by a large language model; the target other modal data is obtained by searching in other modal databases based on the text unit each time the large language model outputs a text unit during the output process of the large language model; the other modal database is constructed based on other modal data in the multimodal data; the modality of the other modal data does not include text; and the output result of the large language model includes at least one text unit.

[0016] In a fourth aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented: In response to user questions, a large language model is called; During the output of the large language model, each time the large language model outputs a text unit, a search is performed in other modality databases based on the text unit; the other modality databases are constructed based on other modality data in the multimodal data; the modality of the other modality data does not include text; the output result of the large language model includes at least one text unit; If the target other modal data is retrieved, assembling the text unit and the target other modal data into a reply to the user's question; If the target other modality data is not retrieved, the output result of the large language model is used as the answer to the user's question.

[0017] In a fifth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented: In response to user questions, a large language model is called; During the output of the large language model, each time the large language model outputs a text unit, a search is performed in other modality databases based on the text unit; the other modality databases are constructed based on other modality data in the multimodal data; the modality of the other modality data does not include text; the output result of the large language model includes at least one text unit; If the target other modal data is retrieved, assembling the text unit and the target other modal data into a reply to the user's question; If the target other modality data is not retrieved, the output result of the large language model is used as the answer to the user's question.

[0018] In a sixth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor: In response to user questions, a large language model is called; During the output of the large language model, each time the large language model outputs a text unit, a search is performed in other modality databases based on the text unit; the other modality databases are constructed based on other modality data in the multimodal data; the modality of the other modality data does not include text; the output result of the large language model includes at least one text unit; If the target other modal data is retrieved, assembling the text unit and the target other modal data into a reply to the user's question; If the target other modality data is not retrieved, the output result of the large language model is used as the answer to the user's question.

[0019] The above-mentioned user question response method, system, computer device and readable storage medium, when the method responds to user questions based on the large language model, each time the large language model outputs a text unit, it searches in other modal databases based on the text unit, and when the target other modal data is retrieved, the text unit and the target other modal data, as well as the subsequent text units and the retrieved target other modal data are assembled into a reply to the user's question. It can be understood that since the target other modal data is retrieved through the text unit output by the large language model, it is strongly correlated with the text unit, thereby ensuring that the text data in the reply and the other modal data have a high correlation, thereby improving the accuracy of the reply output by the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A diagram of an application environment of a user question response method in one embodiment; Figure 2 A flowchart of a method for responding to user questions in one embodiment; Figure 3 A flowchart of a method for responding to user questions in another embodiment; Figure 4 It is a structural block diagram of a user question response device in one embodiment; Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0023] The user question response method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0024] Among them, a large language model is deployed on the server 104, and other modal databases are provided in the data storage system. The user submits a user question to the server 104 through the terminal 102. When the large language model is output, the server 104 searches the data storage system through the text intermediate result output by the large language model, and obtains the answer to the user's question based on the search result and the text intermediate result.

[0025] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart TVs, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0026] As mentioned in the background technology, the responses output by the multimodal large language model are often unsatisfactory. For example, the response contains text data and image data, but the correlation between the text data and the image data is low. The reason is that in traditional technology, multimodal data is generated directly through user questions, but the effect of other modal data (such as pictures, videos, etc.) generated by the current multimodal large language model is poor. For example, if a user asks: How to ride a bicycle, the multimodal large language model can generate a text tutorial and a picture of a person riding a bicycle, but the "bicycle" in the picture may have three wheels. Therefore, the correlation between other modal data and text data generated by the current multimodal large language model is low, resulting in low accuracy of the response.

[0027] In order to solve the problem of low answer accuracy, in an exemplary embodiment, Figure 2 As shown, a method for responding to user questions is provided. When responding to user questions based on a large language model, each time the large language model outputs a text unit, a search is performed in other modal databases based on the text unit, and when the target other modal data is retrieved, the text unit and the target other modal data, as well as subsequent text units and the retrieved target other modal data are assembled into a reply to the user's question. It can be understood that since the target other modal data is retrieved through the text unit output by the large language model, it is strongly correlated with the text unit, thereby ensuring that the text data in the reply and the other modal data have a high degree of correlation, thereby improving the accuracy of the reply output by the large language model.

[0028] This method is applied to Figure 1For example, the server 104 in Figure 2 As shown, the method includes the following steps 202 to 206. Among them: Step 202: In response to a user's question, a large language model is called.

[0029] The user question refers to: during the interaction, the user asks a question to the server 104 through the terminal 102. The user question carries the question submitted by the user. For example, the question is: What is the technology of the large language model? The user question can be input on the interactive interface of the user question response system, that is, the collection interface, and an input box is set in the collection interface for the user to input the question.

[0030] Exemplarily, after receiving the user's question, the server 104 extracts the question contained therein and calls the large language model. That is, the question can be used as the input of the large language model, and the large language model provides a partial answer to the question. It can be understood that the partial answer is text data in the multimodal data.

[0031] Step 204: During the output process of the large language model, each time the large language model outputs a text unit, a search is performed in other modal databases based on the text unit; the other modal databases are constructed based on other modal data in the multimodal data; the modality of the other modal data does not include text; the output result of the large language model includes at least one text unit.

[0032] The text unit includes but is not limited to a sentence, a paragraph, a chapter, and a word.

[0033] Exemplarily, during the output of the large language model, its output result includes at least one text unit, that is, during the search in other modal databases based on the text unit, the search is not performed in other modal databases after the large language model completes the output, but after a text unit is obtained, the search is performed in other modal databases, so as to ensure the timeliness of answering user questions and the similarity between the text unit and the corresponding other modal data. It is understandable that each text unit may retrieve the corresponding other modal data.

[0034] In one embodiment, on the interactive interface of the user question response system, i.e., the answer output interface, it can be reflected that after each text unit output by the large language model, the answer output interface outputs the text unit, and after the text unit, if the corresponding other modal data such as video is retrieved, the other modal data can be output immediately afterwards, so as to facilitate the user to understand the text unit, thereby improving the user experience. After that, the large language model continues to output the next text unit, and the user question response system continues to execute the above process in this embodiment until the large language model completes the output and the search and output of the search results are completed in the other modal databases.

[0035] Of course, in another embodiment, in order to ensure the continuity of the text, although when searching in other modal databases based on text units, a search is performed for each text unit output, after the large language model completes the output, all other corresponding modal data retrieved can be added after its output result, and a reference relationship can be inserted in the corresponding position of each text unit, such as the output end position. For example, the first text unit corresponds to the first other modal data. After the large language model outputs the first text unit, the label "1" can be inserted at the output end position. Correspondingly, in the reply, the label of the first other modal data is 1; for another example, the second text unit corresponds to the second other modal data. After the large language model outputs the second text unit, the label "2" can be inserted at the output end position. Correspondingly, in the reply, the label of the second other modal data is 2; and so on, to obtain the answer to the user's question. It can be understood that this method is similar to the way of citing other articles in a paper. That is, on the reply output interface in the user question response system, the reply can be presented by first displaying the output result of the large language model, inserting a label after each text unit in the output result, and then displaying the corresponding other modal data and its labels.

[0036] It should be noted that before using other modality databases, it is necessary to build the other modality databases.

[0037] In the process of building this other modality database: Exemplarily, other modality databases may be relational databases or vector databases.

[0038] It should be noted that, whether it is a relational database or a vector database, after vectorizing other modal data, the vectorized data and the mapping relationship between the data before vectorization and the data after vectorization (recorded as the third mapping relationship) can be stored in the database. For example, the third mapping relationship can be a source data address such as a file address, or an index, and the file address refers to the address where the other modal data is stored in the file system.

[0039] It can be understood that for searching in a database, the process is to vectorize the text unit, and then perform vector similarity matching from the database based on the vectorized text unit. If the target vector data whose vector similarity reaches the similarity threshold is matched, the source data of other modal data corresponding to the target vector data is found through the third mapping relationship. The source data is used for subsequent assembly to obtain the answer to the user's question. Among them, the similarity threshold can be set as needed, and this embodiment is not limited.

[0040] The process of constructing the other modality database is described below: First, obtain other modal data and build an initial vector database.

[0041] Secondly, other modal data are vectorized.

[0042] For example, if the modalities are images and tables, images can be directly embedded through multimodal pre-trained models (customized artificial intelligence models for embedding) and other models that can implement multimodal embedding to obtain vectorized data, while tables can be converted into images before embedding. Another example is that if the modalities are URLs (Uniform Resource Locators), videos, and voices, for URLs, the corresponding web page content can be summarized in text, and then the summarized text can be vectorized through the multimodal pre-trained model; for videos and voices, text summaries can be performed first, and then vectorization similar to URLs can be performed.

[0043] Afterwards, the vectorized data corresponding to the other modal data are stored in the initial vector database, the other modal data are stored in the file system, and the third mapping relationship is stored in the vector database.

[0044] Step 206: If the target other modal data is retrieved, the text unit and the target other modal data are assembled into a response to the user's question; if the target other modal data is not retrieved, the output result of the large language model is used as the response to the user's question.

[0045] Illustratively, the implementation of the assembly can refer to the aforementioned embodiment, which will not be described in detail here.

[0046] It should be noted that since the multimodal large language model has the ability to output text data and other modal data, and the present embodiment has made corresponding improvements on the basis of the large language model so as to have the capabilities of the multimodal large language model, it can be considered that the present embodiment indirectly realizes the multimodal large language model based on the large language model.

[0047] In an exemplary embodiment, based on the above embodiment, it can also be implemented based on RAG (Retrieval-Augmented Generation) technology.

[0048] In traditional technology, the RAG system implemented based on RAG technology mainly includes three modules, namely, retrieval module, enhancement module and generation module.

[0049] The retrieval module is used to retrieve from the text vector database based on the questions submitted by the user, so as to obtain the information (text data) required by the enhancement module. For example, if the question is "What are the fun places in Hangzhou", the retrieval module may obtain the following text data after retrieval: Recommended famous tourist attractions in Hangzhou: West Lake, Leifeng Pagoda, etc.; What are the attractions worth visiting in Hangzhou: You will definitely regret not going to this place... (content in a travel note). The text vector database is constructed based on the vector data after vectorizing the text data to obtain the vector data.

[0050] Among them, the enhancement module is used to enhance the question based on the information provided by the retrieval module, that is, after assembling the question and information, the input of the large language model is obtained. The response of the large language model to the input will be more in line with the user's needs than the response of the large language model to the question. For example, the question is "What are the fun places in Hangzhou?", and the input can be "The question raised by the user is "What are the fun places in Hangzhou?", please refer to the following information to give an answer: Recommended famous tourist attractions in Hangzhou: West Lake, Leifeng Pagoda, etc.; What are the attractions worth visiting in Hangzhou: You will definitely regret not going to this place..."

[0051] The generation module is used to call the large language model to obtain a response generated by the large language model for the input.

[0052] In summary, the responses generated by the RAG system in traditional technology are text data, and are unable to generate responses containing multimodal data.

[0053] In this embodiment, the generation module is improved. The specific improvements are: In response to a user's question, a search is performed in a text database based on the question carried by the user's question; the text database is constructed based on text data; if the first text data is retrieved, the question and the first text data are assembled as input to a large language model, and the large language model is called; if the first text data is not retrieved, the question is directly used as input to the large language model, and the large language model is called. For this generation module, during the output process of the large language model, each time the large language model outputs a text unit, a search is performed in other modal databases based on the text unit; the output result of the large language model includes at least one text unit; if the target other modal data is retrieved, the text unit and the target other modal data are assembled as a response to the user's question.

[0054] The text database may be a relational database or a vector database. Whether it is a relational database or a vector database, before storing the text data, the text data may be vectorized to obtain vector data. The vector data is then stored in the text database, and the text data corresponding to the vector data may be found during retrieval.

[0055] Optionally, each text unit in the text data can be stored in the text database as a record together with the corresponding vector data. In addition, the text data can also be stored in a file system or other database other than the text database. In this case, the reference address of each text unit needs to be stored in the text database as a record together with the corresponding vector data, wherein the reference address refers to the storage address of the text unit in the file system or other database. For example, if there is a text unit and it is stored in the fifth row of database A, the reference address can be the fifth row of database A.

[0056] It should be noted that in the RAG system, the text data used to construct the text database is organized in the file format of a document. In this embodiment, data organized in the file format of PPT (Microsoft Office PowerPoint, presentation) or text data summarized from other modal data can also be used, thereby expanding the data source and facilitating user use.

[0057] Among them, for any document, there is a certain association between the data of each modality. For example, there is a document whose content is an article. The article involves pictures and texts. It can be understood that the text and pictures generally involve the same theme (or central idea). For example, if the article is a travel guide, the text can be used to describe some characteristics of a recommended tourist spot, and the picture can visualize these characteristics. Therefore, the various modal data in the same document generally involve the same theme, while the themes involved in different documents may be different or have certain differences.

[0058] Among them, the improvement of the generation module is mainly reflected in searching in non-text databases (i.e. other modal databases) through text units, and assembling answers to user questions based on the search results (i.e. target other modal data) and text units. The answers are multimodal data.

[0059] Among them, the implementation of retrieval in the generation module mainly includes the following three methods: Method 1: After obtaining preliminary results through retrieval, the preliminary results are screened to obtain the final retrieval results, i.e., the target other modal data. The screening of the preliminary results is based on the mapping relationship between the text data retrieved from the text database and the other modal data (such as the text data and the other modal data belong to the same document or PPT, or the text data is obtained by summarizing the other modal data) (referred to as the fourth mapping relationship).

[0060] Method 2: When searching, first filter out other modal data to be searched from other modal databases based on the text data retrieved from the text database and the fourth mapping relationship, and then search the other modal data to be searched to obtain the final search result, that is, the target other modal data.

[0061] Method three: when searching, do not pay attention to the fourth mapping relationship, but directly perform a full database search and match in other modal databases.

[0062] Among them, it is worth mentioning that for Method 1 and Method 2, this embodiment emphasizes the fourth mapping relationship, because the target other modal data can be accurately found through the fourth mapping relationship, thereby improving the accuracy of the final output reply. Specifically, taking Method 1, and the case where the text data and other modal data belong to the same document as an example, after retrieving multiple text data from the text database, the candidate document set D corresponding to the text data can be determined first. And after the other modal data to be selected is retrieved from the other modal database, it can be determined whether the other modal data to be selected comes from D. If not, it is discarded. If so, it is determined as the target other modal data, thereby avoiding the low relevance of the target other modal data to the user's question to improve the relevance.

[0063] It should be noted that in order to provide a retrieval basis for the retrieval in the retrieval module and the retrieval in the generation module, it is necessary to first establish a text database and other modal databases, and database construction requires obtaining multimodal data first, and then the database can be built through the multimodal data.

[0064] Among them, multimodal data can be data accumulated by the company that builds the RAG system, or data crawled from the web.

[0065] Among them, the acquired multimodal data may include two situations. Situation one is that the multimodal data is organized in the form of documents or PPTs, and the corresponding modality in this situation generally includes text; Situation two is that the multimodal data is composed of at least one single modality of data, and the corresponding modality in this situation generally does not include text.

[0066] There are two ways to build a database for two situations, namely whether the modality includes text or not: Method 1: In this database construction method, multimodal data is organized in the form of documents or PPTs, that is, the modalities of multimodal data include text.

[0067] In one embodiment, for the method 1 and the method 2, the method 1 includes the following steps: Record a first mapping relationship between the second text data and the third other modal data in the multimodal data (an implementation of the fourth mapping relationship in this embodiment); obtain the text database based on the second text data, and obtain the other modal database based on the third other modal data.

[0068] In another embodiment, for the third approach, the first method includes the following steps: The second text data and the third other modality data are extracted from the document or PPT, and the text database is obtained based on the second text data, and the other modality database is obtained based on the third other modality data.

[0069] Method 2: In the database construction method, the multimodal data consists of data of at least one single modality, that is, the modality of the multimodal data does not include text.

[0070] In one embodiment, for the first and second methods, the multimodal data is recorded as fourth other modal data, and the second method includes the following steps: The third text data is extracted based on the fourth other modal data; a second mapping relationship between the third text data and the fourth other modal data (an implementation of the fourth mapping relationship in this embodiment) is recorded; the text database is obtained based on the third text data, and the other modal database is obtained based on the fourth other modal data.

[0071] Among them, the second mapping relationship can be an association relationship between the third text data and the fourth other modal data. For example, the third text data and the fourth other modal data belonging to the same document or PPT have this association relationship; for example, if the third text data is obtained by textually summarizing the fourth other modal data, it also has this association relationship.

[0072] In another embodiment, with respect to the third approach, the multimodal data is recorded as fourth other modal data, and the second method includes the following steps: The third text data is extracted from the fourth other modality data, and the text database is obtained based on the third text data, and the other modality database is obtained based on the fourth other modality data.

[0073] Among them, for the text data extracted from other modal data involved in method 2, it can be extracted through a multimodal pre-trained model (with the ability to summarize text data from multimodal data) or through a targeted artificial intelligence model. For example, for images, it can be extracted through an image summary model; for tables, the table can be converted into an image and then extracted through an image summary model; for URLs, the web page content corresponding to the URL can be converted into an image and then extracted through an image summary model; for video data, it can be extracted through a video summary model; for speech, it can be extracted through a speech recognition model.

[0074] Exemplarily, after summarizing the other modal data (i.e. extracting the text data), a text database and other modal database can be constructed respectively. And the second mapping relationship is recorded and saved in the database, so that after matching the corresponding text during retrieval, the other modal data can be found through the second mapping relationship and the corresponding text.

[0075] Among them, how to build a database can refer to the database building method above, and will not be repeated here.

[0076] In the process of generating the module for retrieval, there are three different retrieval processes for method 1, method 2 and method 3. Among them: Retrieval process 1: For method one, the method one is implemented through the following steps: if at least one first other modal data is retrieved, then based on the mapping relationship between the text data and the other modal data (i.e., the fourth mapping relationship), data matching is performed from the first other modal data as well as the first text data; if the match is successful, it is determined that the target other modal data is retrieved.

[0077] Exemplarily, if the first other modality data is not retrieved, it is determined that the target other modality data is not retrieved; and if the match fails, it is determined that the target other modality data is not retrieved.

[0078] Retrieval process 2: For method 2, the method 2 is implemented through the following steps: based on the mapping relationship between text data and other modal data (i.e., the fourth mapping relationship), determining the second other modal data corresponding to the first text data for subsequent retrieval from other modal databases; and searching in the second other modal data based on the text unit.

[0079] Retrieval process three: Regarding the third method, the third method is implemented by the following steps: directly searching in other modal databases based on the text unit; if third other modal data is retrieved, determining the third other modal data as the target other modal data.

[0080] In the process of assembling the answer to the user's question by the generation module based on the retrieved other modal data and the text units output by the large language model, the answer to the user's question can be assembled in the following way: Each time a target other modal data is retrieved, the target other modal data is placed between the current text unit and the next text unit until the large language model stops outputting, and an answer to the user's question is obtained.

[0081] Taking the text unit as a paragraph as an example, the answer to the user's question can be assembled in the following way: Each time a target other modal data is retrieved, the target other modal data is placed between the current paragraph and the next paragraph until the large language model stops outputting, and an answer to the user's question is obtained.

[0082] It is worth mentioning that in order to ensure the accuracy of the answer, the vector data in the vector database needs to be formally identical to the text unit. Formally identical means that the text is segmented to obtain the text set corresponding to each vector data in the same way as the text unit is segmented by the output of the large language model. For example, if the text unit is a sentence, then the text set corresponding to each vector data is also a sentence.

[0083] For ease of understanding, Figure 3 An optional embodiment of a method for responding to a user's question is provided, and the embodiment comprises the following steps: Step 1 ( Figure 3 (not shown in the figure): A text database and other modal databases are constructed based on multiple documents, and both the text database and other modal databases are vector databases.

[0084] Step 2: Receive user questions and obtain the questions.

[0085] Step 3: Use this question to search and match in the text vector database (i.e., text database) to obtain multiple paragraphs of text.

[0086] Step 4: Determine the documents corresponding to each paragraph of text and generate a candidate document set D.

[0087] Step 5: Organize the question and multiple paragraphs of text into prompt words, which are the input of the large language model, and call the large language model.

[0088] Step 6: The large language model starts to output answers. During the large language model output process, each sentence it outputs is searched and matched in the non-text vector database (i.e., other modality database).

[0089] Step 7: Retrieve non-text modal data (i.e. other modal data to be selected).

[0090] Step 8: Determine whether the other modal data to be selected belongs to the candidate document set D.

[0091] Step 9: If so, the other modal data to be selected is determined as the target other modal data, and is rendered after the sentence currently output by the large language model as part of the output (i.e., part of the answer) until the large language model completes the output and obtains the answer to the user's question.

[0092] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0093] Based on the same inventive concept, the embodiment of the present application also provides a user question response system, which includes: A collection interface, wherein an input box is provided in the collection interface for the user to input questions; The answer output interface is used to output the answer to the user's question assembled by text units and other target modal data; Wherein, the text unit is output by a large language model; the target other modal data is obtained by searching in other modal databases based on the text unit each time the large language model outputs a text unit during the output process of the large language model; the other modal database is constructed based on other modal data in the multimodal data; the modality of the other modal data does not include text; and the output result of the large language model includes at least one text unit.

[0094] The implementation solution provided by the system to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations can be found in the limitations on the user question response method above, and will not be repeated here.

[0095] Based on the same inventive concept, the embodiment of the present application also provides a user question response device for implementing the user question response method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more user question response device embodiments provided below can refer to the limitations of the user question response method above, and will not be repeated here.

[0096] In an exemplary embodiment, Figure 4 As shown, a user question response device is provided, including: a calling module 402, a retrieval module 404 and an assembling module 406, wherein: A calling module 402 is used to call a large language model in response to a user's question; A retrieval module 404 is used for searching in other modality databases based on each text unit output by the large language model during the output process of the large language model; the other modality database is constructed based on other modality data in the multimodal data; the modality of the other modality data does not include text; the output result of the large language model includes at least one text unit; The assembly module 406 is used to assemble the text unit and the target other modal data into a response to the user's question if the target other modal data is retrieved; if the target other modal data is not retrieved, use the output result of the large language model as the response to the user's question.

[0097] In one embodiment, the calling module 402 includes: A first retrieval unit, configured to respond to a user's question and search a text database based on a question in the user's question; the text database is constructed based on text data; The calling unit is used for assembling the question and the first text data as input of the large language model if the first text data is retrieved, and calling the large language model.

[0098] In one embodiment, the apparatus further comprises: A matching module, configured to perform data matching on the first other modal data based on a mapping relationship between the text data and the other modal data and the first text data if at least one first other modal data is retrieved; The determination module is used to determine whether other modal data of the target is retrieved if the match is successful.

[0099] In one embodiment, the retrieval module 404 includes: A first determining unit, configured to determine, from an other modality database, second other modality data corresponding to the first text data for subsequent retrieval based on a mapping relationship between the text data and the other modality data; A second retrieval unit is used to search the second other modality data based on the text unit.

[0100] In one embodiment, the apparatus further comprises: An acquisition module, used to acquire multimodal data; A first recording module, configured to record a first mapping relationship between second text data and third other modality data in the multimodal data when the modality of the multimodal data includes text; The first database construction module is used to obtain the text database based on the second text data, and to obtain the other modality database based on the third other modality data.

[0101] In one embodiment, the multimodal data is fourth other modality data, and the apparatus further includes: An extraction module, configured to extract third text data based on the fourth other modality data; The second recording module is used for recording a second mapping relationship between the third text data and the fourth other modality data; The second database construction module is used to obtain the text database based on the third text data, and to obtain the other modality database based on the fourth other modality data.

[0102] In one embodiment, the retrieval module 404 includes: A third retrieval unit, used to directly search in other modality databases based on the text unit; The second determining unit is configured to determine that the third other modal data is the target other modal data if the third other modal data is retrieved.

[0103] In one embodiment, the assembling module 406 includes: The assembly unit is used to place a target other modal data between the current text unit and the next text unit each time the target other modal data is retrieved until the large language model stops outputting and the answer to the user's question is obtained.

[0104] Each module in the above-mentioned user question response device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0105] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store multimodal data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a user question response method is implemented.

[0106] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0107] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented: In response to user questions, a large language model is called; During the output of the large language model, each time the large language model outputs a text unit, a search is performed in other modality databases based on the text unit; the other modality databases are constructed based on other modality data in the multimodal data; the modality of the other modality data does not include text; the output result of the large language model includes at least one text unit; If the target other modal data is retrieved, assembling the text unit and the target other modal data into a reply to the user's question; If the target other modality data is not retrieved, the output result of the large language model is used as the answer to the user's question.

[0108] In one embodiment, when the processor executes the computer program, the processor further implements the following steps: In response to a user's question, searching a text database based on the question carried in the user's question; the text database is constructed based on text data; If the first text data is retrieved, the question and the first text data are assembled as input of a large language model, and the large language model is called.

[0109] In one embodiment, when the processor executes the computer program, the processor further implements the following steps: If at least one first other modal data is retrieved, data matching is performed on the first other modal data based on the mapping relationship between the text data and the other modal data and the first text data; If the match is successful, it is determined that the target other modal data is retrieved.

[0110] In one embodiment, when the processor executes the computer program, the processor further implements the following steps: Based on the mapping relationship between the text data and the other modal data, determining second other modal data corresponding to the first text data for subsequent retrieval from the other modal database; A search is performed in the second other modality data based on the text unit.

[0111] In one embodiment, when the processor executes the computer program, the processor further implements the following steps: Acquire multimodal data; In a case where the modality of the multimodal data includes text, recording a first mapping relationship between second text data and third other modality data in the multimodal data; The text database is obtained based on the second text data, and the other modality database is obtained based on the third other modality data.

[0112] In one embodiment, when the processor executes the computer program, the following steps are also implemented: The multimodal data is fourth other modal data, and the third text data is extracted based on the fourth other modal data; Recording a second mapping relationship between the third text data and the fourth other modality data; The text database is obtained based on the third text data, and the other modality database is obtained based on the fourth other modality data.

[0113] In one embodiment, when the processor executes the computer program, the following steps are also implemented: Directly search in other modality databases based on the text unit; If the third other modal data is retrieved, the third other modal data is determined to be the target other modal data.

[0114] In one embodiment, when the processor executes the computer program, the following steps are also implemented: Each time a target other modal data is retrieved, the target other modal data is placed between the current text unit and the next text unit until the large language model stops outputting, and an answer to the user's question is obtained.

[0115] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: In response to user questions, a large language model is called; During the output of the large language model, each time the large language model outputs a text unit, a search is performed in other modality databases based on the text unit; the other modality databases are constructed based on other modality data in the multimodal data; the modality of the other modality data does not include text; the output result of the large language model includes at least one text unit; If the target other modal data is retrieved, assembling the text unit and the target other modal data into a reply to the user's question; If the target other modality data is not retrieved, the output result of the large language model is used as the answer to the user's question.

[0116] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: In response to a user's question, searching a text database based on the question carried in the user's question; the text database is constructed based on text data; If the first text data is retrieved, the question and the first text data are assembled as input of a large language model, and the large language model is called.

[0117] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: If at least one first other modal data is retrieved, data matching is performed on the first other modal data based on the mapping relationship between the text data and the other modal data and the first text data; If the match is successful, it is determined that the target other modal data is retrieved.

[0118] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: Based on the mapping relationship between the text data and the other modal data, determining second other modal data corresponding to the first text data for subsequent retrieval from the other modal database; A search is performed in the second other modality data based on the text unit.

[0119] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: Acquire multimodal data; In a case where the modality of the multimodal data includes text, recording a first mapping relationship between second text data and third other modality data in the multimodal data; The text database is obtained based on the second text data, and the other modality database is obtained based on the third other modality data.

[0120] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: The multimodal data is fourth other modal data, and the third text data is extracted based on the fourth other modal data; Recording a second mapping relationship between the third text data and the fourth other modality data; The text database is obtained based on the third text data, and the other modality database is obtained based on the fourth other modality data.

[0121] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: Directly search in other modality databases based on the text unit; If the third other modal data is retrieved, the third other modal data is determined to be the target other modal data.

[0122] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: Each time a target other modal data is retrieved, the target other modal data is placed between the current text unit and the next text unit until the large language model stops outputting, and an answer to the user's question is obtained.

[0123] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps: In response to user questions, a large language model is called; During the output of the large language model, each time the large language model outputs a text unit, a search is performed in other modality databases based on the text unit; the other modality databases are constructed based on other modality data in the multimodal data; the modality of the other modality data does not include text; the output result of the large language model includes at least one text unit; If the target other modal data is retrieved, assembling the text unit and the target other modal data into a reply to the user's question; If the target other modality data is not retrieved, the output result of the large language model is used as the answer to the user's question.

[0124] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: In response to a user's question, searching a text database based on the question carried in the user's question; the text database is constructed based on text data; If the first text data is retrieved, the question and the first text data are assembled as input of a large language model, and the large language model is called.

[0125] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: If at least one first other modal data is retrieved, data matching is performed on the first other modal data based on the mapping relationship between the text data and the other modal data and the first text data; If the match is successful, it is determined that the target other modal data is retrieved.

[0126] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: Based on the mapping relationship between the text data and the other modal data, determining second other modal data corresponding to the first text data for subsequent retrieval from the other modal database; A search is performed in the second other modality data based on the text unit.

[0127] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: Acquire multimodal data; In a case where the modality of the multimodal data includes text, recording a first mapping relationship between second text data and third other modality data in the multimodal data; The text database is obtained based on the second text data, and the other modality database is obtained based on the third other modality data.

[0128] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: The multimodal data is fourth other modal data, and the third text data is extracted based on the fourth other modal data; Recording a second mapping relationship between the third text data and the fourth other modality data; The text database is obtained based on the third text data, and the other modality database is obtained based on the fourth other modality data.

[0129] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: Directly search in other modality databases based on the text unit; If the third other modal data is retrieved, the third other modal data is determined to be the target other modal data.

[0130] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: Each time a target other modal data is retrieved, the target other modal data is placed between the current text unit and the next text unit until the large language model stops outputting, and an answer to the user's question is obtained.

[0131] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0132] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0133] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0134] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A user question response method, characterized in that: The method comprises: In response to user questions, a large language model is called; During the output of the large language model, each time the large language model outputs a text unit, a search is performed in other modality databases based on the text unit; the other modality databases are constructed based on other modality data in the multimodal data; the modality of the other modality data does not include text; the output result of the large language model includes at least one text unit; If the target other modal data is retrieved, assembling the text unit and the target other modal data into a reply to the user's question; If the target other modality data is not retrieved, the output result of the large language model is used as the answer to the user's question.

2. The method according to claim 1, characterized in that In response to the user's question, calling the large language model includes: In response to a user's question, searching a text database based on the question in the user's question; the text database is constructed based on text data; If the first text data is retrieved, the question and the first text data are assembled as input of a large language model, and the large language model is called.

3. The method according to claim 2, characterized in that After searching in other modality databases based on the text unit, the method includes: If at least one first other modal data is retrieved, data matching is performed on the first other modal data based on the mapping relationship between the text data and the other modal data and the first text data; If the match is successful, it is determined that the target other modal data is retrieved.

4. The method according to claim 2, characterized in that: The searching in other modality databases based on the text unit includes: Based on the mapping relationship between the text data and the other modal data, determining second other modal data corresponding to the first text data for subsequent retrieval from the other modal database; A search is performed in the second other modality data based on the text unit.

5. The method according to claim 3 or claim 4, characterized in that: The method further comprises: Acquire multimodal data; In a case where the modality of the multimodal data includes text, recording a first mapping relationship between second text data and third other modality data in the multimodal data; The text database is obtained based on the second text data, and the other modality database is obtained based on the third other modality data.

6. The method according to claim 5, characterized in that After the multimodal data is obtained, the method further includes: The multimodal data is fourth other modal data, and the third text data is extracted based on the fourth other modal data; Recording a second mapping relationship between the third text data and the fourth other modality data; The text database is obtained based on the third text data, and the other modality database is obtained based on the fourth other modality data.

7. The method according to claim 2, characterized in that: The searching in other modality databases based on the text unit includes: Directly search in other modality databases based on the text unit; If the third other modal data is retrieved, the third other modal data is determined to be the target other modal data.

8. The method according to claim 1, characterized in that If the target other modal data is retrieved, assembling the text unit and the target other modal data into a reply to the user's question includes: Each time a target other modal data is retrieved, the target other modal data is placed between the current text unit and the next text unit until the large language model stops outputting, and an answer to the user's question is obtained.

9. A user question response system, characterized in that: The system comprises: A collection interface, wherein an input box is provided in the collection interface for the user to input questions; The answer output interface is used to output the answer to the user's question assembled by the text unit and other target modal data; Wherein, the text unit is output by a large language model; the target other modal data is obtained by searching in other modal databases based on the text unit each time the large language model outputs a text unit during the output process of the large language model; the other modal database is constructed based on other modal data in the multimodal data; the modality of the other modal data does not include text; and the output result of the large language model includes at least one text unit.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images