Intelligent question answering method, device and equipment based on large model, intelligent agent and medium

Abstract

The invention provides an intelligent question and answer method based on a large model, and relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like. According to the specific implementation scheme, under the condition that it is determined that target questions are in a multi-mode form, non-text questions in the target questions are recognized, a recognition result is obtained, and the target questions further comprise text questions; rewriting the text problem by using the recognition result to obtain a rewritten text problem; and inputting the rewritten text question, the text question and the non-text question into the large model to obtain a target answer. The invention provides an intelligent question answering device based on a large model, electronic equipment, an intelligent agent, a medium and a product.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to computer vision, deep learning, large models and other technical fields, and specifically to an intelligent question-answering method, device, electronic device, intelligent agent, storage medium and program product based on a large model. Background Art

[0002] With the rapid advancement of deep learning technology, large model products have gradually become the mainstay in the field of intelligent dialogue services. Based on massive data training, large models can capture complex language patterns and user intentions, providing users with personalized and intelligent dialogue experiences. However, how to make large models accurately understand the true intentions of users has become a research focus. Summary of the invention

[0003] The present invention provides a large-model-based intelligent question-answering method, device, electronic device, storage medium, and program product.

[0004] According to one aspect of the present disclosure, a smart question-answering method based on a large model is provided, comprising: when it is determined that a target question is in a multimodal form, identifying a non-text question in the target question to obtain a recognition result, wherein the target question also includes a text question; using the recognition result to rewrite the text question to obtain a rewritten text question; and inputting the rewritten text question, the text question and the non-text question into the large model to obtain a target answer.

[0005] According to another aspect of the present disclosure, an intelligent question-answering device based on a large model is provided, comprising: a recognition module, for recognizing non-text questions in a target question when it is determined that the target question is in a multimodal form, and obtaining a recognition result, wherein the target question also includes a text question; a rewriting module, for rewriting the text question using the recognition result, and obtaining a rewritten text question; and a processing module, for inputting the rewritten text question, the text question and the non-text question into the large model, and obtaining a target answer.

[0006] According to another aspect of the present disclosure, there is provided an artificial intelligence agent, comprising: an input module for receiving input information; a processing module for determining a target task based on the input information received by the input module, determining a large language model based on the target task, and executing the above method by calling the large language model; and an output module for outputting output information obtained by the processing module.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method as disclosed in the present disclosure.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method of the present disclosure.

[0009] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, and when the computer program is executed by a processor, the computer program implements the method of the present disclosure.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0012] Figure 1 An exemplary system architecture to which a large-model-based intelligent question-answering method and device can be applied according to an embodiment of the present disclosure is schematically shown;

[0013] Figure 2 The flowchart of the intelligent question-answering method based on a large model according to an embodiment of the present disclosure is schematically shown;

[0014] Figure 3 The effect diagram of determining a multimodal target problem according to an embodiment of the present disclosure is schematically shown;

[0015] Figure 4 A flowchart for determining a recognition result according to an embodiment of the present disclosure is schematically shown;

[0016] Figure 5 A flowchart of an intelligent question-answering method based on a large model according to another embodiment of the present disclosure is schematically shown;

[0017] Figure 6 The block diagram schematically shows an intelligent question-answering device based on a large model according to an embodiment of the present disclosure.

[0018] Figure 7 A block diagram schematically shows a structure of an artificial intelligence agent according to an embodiment of the present disclosure; and

[0019] Figure 8A block diagram of an electronic device suitable for implementing a large model-based intelligent question-answering method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0020] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0021] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0022] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0023] Figure 1 An exemplary system architecture to which a large-model-based intelligent question-answering method and apparatus can be applied according to an embodiment of the present disclosure is schematically shown.

[0024] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, an exemplary system architecture to which the large-model-based intelligent question-answering method and apparatus can be applied may include a terminal device, but the terminal device may implement the large-model-based intelligent question-answering method and apparatus provided in the embodiments of the present disclosure without interacting with the server.

[0025] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0026] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as examples).

[0027] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0028] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for the content browsed by the user using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device.

[0029] It should be noted that the intelligent question-answering method based on a large model provided in the embodiment of the present disclosure can generally be executed by the terminal device 101, 102, or 103. Accordingly, the intelligent question-answering device based on a large model provided in the embodiment of the present disclosure can also be set in the terminal device 101, 102, or 103.

[0030] Alternatively, the intelligent question-answering method based on a large model provided in the embodiment of the present disclosure may also be generally executed by the server 105. Accordingly, the intelligent question-answering device based on a large model provided in the embodiment of the present disclosure may generally be arranged in the server 105. The intelligent question-answering method based on a large model provided in the embodiment of the present disclosure may also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the intelligent question-answering device based on a large model provided in the embodiment of the present disclosure may also be arranged in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0031] For example, when a user inputs a target question in the interactive interface, the terminal devices 101, 102, and 103 may obtain the target question input by the user, and then send the obtained target question to the server 105, which recognizes and rewrites the target question and outputs the target answer. Alternatively, a server or server cluster that can communicate with the terminal devices 101, 102, 103 and / or the server 105 may recognize and rewrite the target question and output the target answer.

[0032] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0033] It should be noted that the sequence numbers of the operations in the following method are only used as representations of the operations for the purpose of description, and should not be regarded as representing the execution order of the operations. Unless explicitly stated, the method does not need to be executed completely in the order shown.

[0034] In some question-and-answer scenarios based on big models, users may combine the content of text modality and other modalities to ask questions to the big model. The input text may be relatively brief, making it difficult for the big model to understand the user's true intentions and failing to correctly stimulate the knowledge of the big model, resulting in inaccurate responses to questions.

[0035] The present disclosure provides an intelligent question-answering method based on a large model. When a target question is determined to be in a multimodal form, a non-text question in the target question is identified to obtain a recognition result. The target question also includes a text question. The text question is rewritten using the recognition result to obtain a rewritten text question. The rewritten text question, the text question, and the non-text question are input into the large model to obtain a target answer.

[0036] When the target question is in multimodal form, non-text questions are identified and the text questions are rewritten using the identification results to obtain more comprehensive and accurate rewritten text questions, making the rewritten text questions more in line with user intentions. Subsequently, the rewritten text questions, original text questions, and non-text questions are input into the large model for processing, which promotes the effective integration and complementarity of text questions, non-text questions, and rewritten text questions, thereby improving the accuracy and relevance of the answers to the questions.

[0037] Figure 2 The flowchart of the intelligent question-answering method based on a large model according to an embodiment of the present disclosure is schematically shown.

[0038] like Figure 2 As shown, the method includes operations S210 to S230.

[0039] In operation S210, when it is determined that the target question is in a multimodal form, a non-text question in the target question is identified to obtain a recognition result, and the target question also includes a text question.

[0040] In operation S220, the text question is rewritten using the recognition result to obtain a rewritten text question.

[0041] In operation S230, the rewritten text question, the text question, and the non-text question are input into the large model to obtain a target answer.

[0042] In an embodiment of the present disclosure, the multimodal form may include a text modal form and a non-text modal form. Correspondingly, the target question may consist of a text question and a non-text question. The non-text question represents information or data in a non-text modal form. The text question may be a question in a text modal form. As a component of the target question, the text question is usually a relatively brief, broad or vague question.

[0043] The non-text modal forms may include modal forms such as images, audio, and video. The target question in a multimodal form may be a combination of a text question and at least one of the modalities of images, audio, and video. For example, the target question in a multimodal form may be a combination of a text question and one or more images, a combination of a text question and video, a combination of a text question and audio, a combination of a text question and audio and images, and so on.

[0044] In some embodiments, the text questions and non-text questions may be related in content. The text questions may provide context, explanation, or guidance to the non-text questions; or the non-text questions may provide context, explanation, or guidance to the text questions.

[0045] In some embodiments, the text question and the non-text question can complement each other and together constitute a complete description of the question. For example, a text question is "What song can this play?", and the corresponding non-text question is a picture of a musical instrument.

[0046] After the target question is determined to be in multimodal form, the type of non-text modality can be determined first, and then a matching method can be selected to identify the non-text question according to the type of non-text modality. For example, images can be identified using image recognition technology, audio can be identified using speech recognition technology, and videos can be identified using video analysis technology.

[0047] The recognition result represents the key feature information of the non-text problem. For example, if the non-text modality is an image, the recognition result can be the object, scene or face in the image. If the non-text modality is audio, the recognition result can be the speech content, speaker identity or emotion. If the non-text modality is a video, the recognition result can be the key information, action, event, object and the relationship between them in the video.

[0048] Rewriting the text question using the recognition result may be combining the recognition result with the text question to rewrite the text question and obtain a more complete and specific rewritten text question, thereby making the rewritten text question more in line with the user's intention.

[0049] In some embodiments, the recognition result is combined with the text question, and the text question is rewritten. The text question can be reconstructed using the recognition result to obtain the rewritten text question. The present disclosure is not limited to this, and the recognition result can also be added to the text question as supplementary information, scenario, condition or constraint to obtain the rewritten text question. It can also be that the original text question is decomposed into multiple sub-questions, each sub-question is associated with an aspect of the recognition result, etc., and the rewritten text question is obtained according to the multiple sub-questions.

[0050] After rewriting the text question to obtain the rewritten text question, the rewritten text question, the text question and the non-text question are input into the big model. The big model can retrieve relevant knowledge based on the rewritten text question, the text question and the non-text question, and obtain the target answer based on the retrieved relevant knowledge.

[0051] A large model may refer to a deep learning model with large-scale model parameters. A large model usually contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than 10 trillion model parameters. The large model involved in the embodiments of the present disclosure may be a general large model, or may be an expert large model obtained after fine-tuning (Fine Tune) based on demand, and the embodiments of the present disclosure are not limited to this. The large model can be used to process information of any modality such as images, text, video, audio, etc.

[0052] According to the embodiments of the present disclosure, when the target question is in a multimodal form, a more comprehensive and accurate rewritten text question is obtained by identifying the non-text question and rewriting the text question using the identification result, and the rewritten text question is more in line with the user's intention. The rewritten text question, the original text question and the non-text question are input into the large model for processing, which promotes the effective integration and complementarity of the text question, the non-text question and the rewritten text question, thereby improving the accuracy and relevance of the question answering.

[0053] According to an embodiment of the present disclosure, when there is a reference relationship between a text question and a non-text question, the text question and the non-text question are determined to be target questions in a multimodal form. The reference relationship between the text question and the non-text question is obtained through a reference control identifier of an interactive interface.

[0054] A reference control indicator can be an element or mechanism on an interactive interface that is used to clarify the reference relationship between a text question and a non-text question. The reference control indicator can indicate the relationship between the two through a label, a hyperlink, or an embedded object.

[0055] In some embodiments, in response to the user's operation on the interactive information in the interactive interface, a reference control may be displayed at the location of the interactive information, and the interactive information may include text questions, non-text questions, and target answers, etc. The user may reference the interactive information by clicking on the reference control, a reference control mark is formed at the input location of the current text question, and the non-text content is referenced.

[0056] In some embodiments, the text question and the non-text question may be interactive information sent at different times in the interactive interface. When the user inputs a text question, the user may reference the non-text question through a reference control, and display the text question, the non-text question, and the target question identified by the reference control at the current input position.

[0057] Figure 3 The effect diagram of determining a multimodal target problem according to an embodiment of the present disclosure is schematically shown.

[0058] like Figure 3 As shown, the user inputs a non-text question 301 containing a picture at the first time. After several rounds of interaction, the user can input a text question 302 at the second time and quote the non-text question 301 again through the reference control. A reference control logo 303 and the quoted non-text question 301 are displayed below the text question 302. By identifying the reference control logo 303, it can be determined that there is a reference relationship between the text question 302 and the non-text question 301, and then it is determined that the target question input this time is in a multimodal form.

[0059] In some embodiments, the target question can also be determined to be a target question in a multimodal form based on the input time and position relationship between the text question and the non-text question, or the reference information to the non-text question contained in the text question. For example, the user inputs a text question and a non-text question at the same time. At this time, it can be determined that the current input is a target question in a multimodal form. Alternatively, the text question and the non-text question are in adjacent positions, and it is determined that there is a correlation between the text question and the non-text question, thereby determining that the current input is a target question in a multimodal form. Alternatively, the text question contains clear reference information, such as "above picture", "previous picture" and other information, which can determine that the current input is a target question in a multimodal form.

[0060] According to an embodiment of the present disclosure, by determining the reference relationship between text questions and non-text questions through reference control identifiers, the associated text questions and non-text questions can be quickly located, the coherence between interactive information can be maintained, and the accuracy and efficiency of interactive information processing can be improved.

[0061] According to an embodiment of the present disclosure, when it is determined that the target question is in a multimodal form, identifying the non-text question in the target question to obtain the recognition result may include: performing text element recognition and image element recognition on the non-text question respectively to obtain multiple initial recognition results. Optimizing the multiple initial recognition results to obtain the recognition result.

[0062] In the embodiments of the present disclosure, the non-text modality may be an image, video, or other modality.

[0063] For non-text questions, text element recognition can be used to identify words or numbers in an image. For non-text questions, image element recognition can be used to identify specific objects in an image, as well as the object's action, shape, color, and spatial relationship.

[0064] Multiple visual models can be used to identify multiple text elements and image elements in non-text questions. For example, text elements in non-text questions can be identified by a text recognition model, plant elements in non-text questions can be identified by a plant recognition model, animal elements in non-text questions can be identified by an animal recognition model, and other subject elements in non-text questions other than animal elements or plant elements can be identified by a subject recognition model, etc. By using multiple visual models for parallel processing, the recognition accuracy can be improved while the processing efficiency can be improved.

[0065] Each recognized text element or image element may be taken as an initial recognition result, and multiple recognition results may be obtained based on the recognized text elements and image elements.

[0066] In some embodiments, optimizing multiple initial recognition results to obtain recognition results may include correcting the multiple initial recognition results. For example, the text recognition results may be corrected to correct grammatical errors or spelling errors.

[0067] In some embodiments, optimizing the multiple initial recognition results to obtain the recognition result may include screening the multiple initial recognition results to remove noise information.

[0068] In some embodiments, optimizing multiple initial recognition results to obtain a recognition result may include fusing the multiple initial recognition results, and the obtained recognition result may be a description of a non-text problem determined based on text elements and image elements.

[0069] According to the embodiments of the present disclosure, by simultaneously recognizing text elements and image elements of non-text questions, the information in the non-text questions can be fully captured, and multiple initial recognition results can be optimized to further reduce recognition errors and noise and improve recognition accuracy.

[0070] According to an embodiment of the present disclosure, optimizing the multiple initial recognition results to obtain the recognition result may also include: when the multiple initial recognition results have the same content, performing deduplication processing on the same content to obtain the recognition result. When the multiple initial recognition results do not have the same content, obtaining the recognition result based on the multiple initial recognition results.

[0071] The existence of the same content in multiple initial recognition results may be due to the following reasons, including the existence of repeated elements in non-text problems. For example, a certain word or symbol appears multiple times in the image, resulting in the same content in the initial recognition results. Alternatively, an element may be recognized as multiple identical contents due to errors in the recognition algorithm, resulting in the same content in the initial recognition results. Alternatively, during the recognition process, non-text problems are processed in parallel by using multiple visual models, so that a partial element is processed by multiple visual models at the same time, resulting in the same content in the initial recognition results, etc. Duplicate content can cause redundancy and confusion in the recognition results, so deduplication is required.

[0072] Multiple initial recognition results of the same category can be compared for similarity, and when the similarity is greater than a preset threshold, it can be considered that there are duplicate contents in the multiple initial recognition results. When it is determined that there are identical contents in the multiple initial recognition results, redundant duplicate contents are deleted from the initial recognition results to obtain a recognition result.

[0073] If the same content does not exist in the multiple initial recognition results, the multiple initial recognition results can be directly used as the recognition result, or the multiple initial recognition results can be concatenated or fused to obtain the recognition result.

[0074] According to an embodiment of the present disclosure, when multiple initial recognition results have the same content, redundancy and confusion can be avoided by eliminating duplicate recognition results through deduplication processing, thereby improving recognition accuracy.

[0075] Figure 4 The flowchart of determining the recognition result according to the embodiment of the present disclosure is schematically shown.

[0076] like Figure 4 As shown, the text element 402 and the image element recognition are respectively performed on the non-text question 401, wherein the image element recognition includes the recognition of the animal element 403, the plant element 404 and the other main elements 405, and a plurality of initial recognition results 406 are obtained, and the plurality of initial recognition results 406 are optimized to obtain the recognition result 407. The optimized recognition result 407 can not only fully capture the information in the non-text question, but also further reduce the recognition errors and noise through the optimization process, thereby improving the recognition accuracy.

[0077] According to an embodiment of the present disclosure, rewriting the text question using the recognition result to obtain the rewritten text question may include: generating thought chain prompt information based on the text question and the recognition result. The thought chain prompt information is used to instruct the question rewriting model to perform operations in a thought chain manner. The thought chain prompt information is input into the question rewriting model to obtain the rewritten text question.

[0078] The chain thinking method represents a structured thinking method. The chain thinking method can decompose a task into a series of interrelated and gradually deepening steps. Each step is a necessary link to perform the task, and each step is arranged in a specific logical order.

[0079] In some embodiments, after obtaining the recognition result of the non-text question, the correlation between the recognition result and the text question can be analyzed. For example, it can be understood how the object, scene or action in the image is related to the text question, or how the information in the audio or video supplements or affects the understanding of the text question.

[0080] According to the association between the recognition result and the text question, a thought chain prompt information can be generated. The thought chain prompt information can be presented in the form of natural language and can contain specific tags or structures to instruct the large model to perform operations in a thought chain manner.

[0081] For example, the text question is "What are the animals in the picture doing?" and the non-text question is a picture of a cat chasing a mouse. The non-text question is recognized and the recognition result is obtained: the image contains a cat (object) and a mouse (object), and the cat is chasing the mouse (action). The text question is understood and it is determined that the text question asks about the behavior of the animals in the picture. The association between the recognition result and the text question is analyzed to determine that the animals in the image are a cat and a mouse, and the behavior between them is chasing. Based on the association between the recognition result and the text question, a chain of thought prompt information is generated, for example: first recognize the animals in the image, then determine the behavioral relationship between them, and finally rewrite the text question based on this relationship. The above chain of thought prompt information is input into the question rewriting model. Based on the guidance of the prompt information, the question rewriting model generates a rewritten text question, such as "What is the cat in the picture chasing?" or "What action is the cat in the picture doing?".

[0082] According to the embodiments of the present disclosure, by generating thought chain prompt information based on the recognition results and text questions, the problem rewriting model can be guided to rewrite the text questions in a logically clear and step-by-step manner, thereby achieving accurate rewriting of the text questions and improving the accuracy of target problem processing.

[0083] According to an embodiment of the present disclosure, rewriting the text question using the recognition result to obtain the rewritten text question may include: extracting keywords of a preset type from the recognition result, and replacing pronouns in the text question with the keywords to obtain the rewritten text question.

[0084] The key subjects in the non-text questions of the preset types of keywords can be keywords such as names of people, places, time, events, etc. The text questions may contain some pronouns, such as "he", "she", "it", "here", "there", etc. The specific references of these pronouns are not clear enough, resulting in the large model not being able to accurately identify the user's intention. By replacing the keywords extracted from the recognition results with the corresponding pronoun positions in the text questions, a more specific and clear rewritten text question is generated.

[0085] For example, the text question is "What song can this play?" and the non-text question is a picture of a ukulele. By recognizing the picture, the recognition result is "ukulele", and then "ukulele" is used to replace "this" in the text question, and the rewritten text question is "What song can the ukulele play?".

[0086] According to an embodiment of the present disclosure, by replacing relevant pronouns with keywords of preset types, the rewritten text question can be made more specific and clear, thereby facilitating subsequent answers or processing.

[0087] According to an embodiment of the present disclosure, the big model includes a multimodal big model and a fusion big model. Inputting the rewritten text question, the text question, and the non-text question into the big model to obtain the target answer may include: inputting the rewritten text question, the text question, and the non-text question into the multimodal big model to obtain multiple initial answers. Inputting the multiple initial answers into the fusion big model to obtain the target answer.

[0088] Multimodal large models are able to understand and process information from different modalities. Multimodal large models can internally integrate information from different modalities. For example, when text questions and non-text questions are input at the same time, the model can combine text questions and non-text questions to generate answers.

[0089] The fusion big model can receive multiple initial answers from the multimodal big model, and through further analysis and reasoning, integrate the useful information in the initial answers, eliminate redundant or erroneous parts, and thus generate a more refined and accurate final answer.

[0090] In some embodiments, a training data set of a multimodal large model such as a text question and images, videos, and audios associated with the text question may be collected to train the initial multimodal large model to obtain a multimodal large model. The initial answer generated by the initial multimodal large model may be used as a training data set of a fusion large model to train the initial fusion large model to obtain a fusion large model.

[0091] In some embodiments, inputting the rewritten text question, the text question, and the non-text question into the multimodal big model to obtain multiple initial answers may include: inputting the rewritten text question, the text question, and the non-text question into the multimodal big model, using the multimodal big model to process the text question and the non-text question to obtain a first initial answer; using the multimodal big model to process the rewritten text question and the non-text question in combination to obtain a second initial answer. Inputting the first initial answer and the second initial answer into the fusion big model to obtain a target answer.

[0092] According to the embodiments of the present disclosure, since the rewritten text question may also be inaccurate, the initial answer obtained based on the rewritten text question may be inaccurate. By inputting the text question, non-text question and rewritten text question into the multimodal large model, multiple initial answers are obtained. After the multiple initial answers are fused using the fusion large model, the inaccurate initial answers can be corrected, thereby obtaining a target answer that is closer to the actual situation.

[0093] Figure 5 A flowchart of an intelligent question-answering method based on a large model according to another embodiment of the present disclosure is schematically shown.

[0094] like Figure 5 As described above, when it is determined that the target question is in a multimodal form, the non-text question 502 in the target question is identified to obtain an identification result 503. Based on the identification result 503 and the text question 501, a thought chain prompt information 504 is generated, and the thought chain prompt information 504 is input into the question rewriting model 505 to obtain a rewritten text question 506. The rewritten text question 506 is not only more comprehensive and accurate but also more in line with the user's intention. The text question 501, the non-text question 502 and the rewritten text question 506 are input into the multimodal model 507 to obtain multiple initial recognition results, and the multiple initial recognition results are input into the fusion model 508 to obtain the target answer 509. The obtained target answer 509 integrates all the valid information of the text question 501, the non-text question 502 and the rewritten text question 506, thereby improving the accuracy and relevance of the question answering.

[0095] Figure 6 A block diagram of an intelligent question-answering device based on a large model according to an embodiment of the present disclosure is schematically shown.

[0096] like Figure 6 As shown, the device includes an identification module 610 , a rewriting module 620 and a processing module 630 .

[0097] The recognition module 610 is used to recognize non-text questions in the target question when it is determined that the target question is in a multimodal form, and obtain a recognition result. The target question also includes a text question.

[0098] The rewriting module 620 is used to rewrite the text question using the recognition result to obtain a rewritten text question.

[0099] The processing module 630 is used to input the rewritten text questions, text questions and non-text questions into the big model to obtain the target answer.

[0100] According to an embodiment of the present disclosure, the rewriting module 620 includes a generating submodule and a rewriting submodule.

[0101] The generation submodule is used to generate thought chain prompt information based on the text question and the recognition result, wherein the thought chain prompt information is used to instruct the question rewriting model to perform operations in a thought chain manner.

[0102] The rewriting submodule is used to input the thought chain prompt information into the problem rewriting model to obtain the rewriting text problem.

[0103] According to an embodiment of the present disclosure, the rewriting module 620 further includes an extraction submodule and a replacement submodule.

[0104] An extraction submodule, used to extract keywords of preset types from the recognition results; and

[0105] The replacement submodule is used to replace the pronouns in the text question with keywords to obtain the rewritten text question.

[0106] According to an embodiment of the present disclosure, the large model includes a multimodal large model and a fusion large model. The processing module 630 includes a processing submodule and a fusion submodule.

[0107] The processing submodule is used to input the rewritten text questions, text questions and non-text questions into the multimodal large model to obtain multiple initial answers.

[0108] The fusion submodule is used to input multiple initial answers into the fusion model to obtain the target answer.

[0109] According to an embodiment of the present disclosure, the identification module 610 includes an identification submodule and an optimization submodule.

[0110] A recognition submodule, for performing text element recognition and image element recognition on non-text questions to obtain multiple initial recognition results; and

[0111] The optimization submodule is used to optimize the multiple initial recognition results to obtain the recognition results.

[0112] According to an embodiment of the present disclosure, the optimization submodule includes a deduplication unit and a determination unit.

[0113] a deduplication unit, configured to, when the same content exists in a plurality of initial recognition results, perform deduplication processing on the same content to obtain a recognition result; and

[0114] The determination unit is used to obtain a recognition result based on the multiple initial recognition results when the multiple initial recognition results do not have the same content.

[0115] According to an embodiment of the present disclosure, the device further includes a determination module.

[0116] The determination module is used to determine that the text question and the non-text question are target questions in a multimodal form when there is a reference relationship between the text question and the non-text question. The reference relationship between the text question and the non-text question is obtained through a reference control identifier of the interactive interface.

[0117] According to an embodiment of the present disclosure, any multiple modules of the identification module 610, the rewriting module 620 and the processing module 630 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the identification module 610, the rewriting module 620 and the processing module 630 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the identification module 610, the rewriting module 620 and the processing module 630 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0118] Figure 7 The structural block diagram of an artificial intelligence agent according to an embodiment of the present disclosure is schematically shown.

[0119] In the embodiments of the present disclosure, Figure 7 As shown, the AI ​​agent 700 may include an input module 710 , a processing module 720 , and an output module 730 .

[0120] An input module 710, for receiving input information;

[0121] The processing module 720 is used to determine the target task based on the input information received by the input module, determine the big model based on the target task, and execute the big model-based intelligent question-answering method provided according to the embodiment of the present disclosure by calling the big model.

[0122] The output module 730 is used to output the output information obtained by the processing module.

[0123] According to an embodiment of the present disclosure, the input module 710 is responsible for receiving or sensing information such as queries, requests, instructions, signals or data from the outside world (e.g., users or external environment), and converting them into a format that can be understood and processed by the AI ​​agent 700. The input module 710 is the primary link for the AI ​​agent 800 to interact with the outside world, which enables the AI ​​agent 700 to efficiently and accurately obtain the necessary "sensory" information from the outside world and respond to this information.

[0124] In an example, the input module 710 may input the target question, rewritten text question, etc. described above.

[0125] In the example, the processing module 720 is the core support for the AI ​​agent 700 to process complex tasks. The processing module 720 can execute the intelligent question-answering method based on the large model described above.

[0126] In the example, the performance of the processing module 720 may be closely related to the large model on which the AI ​​agent 700 is based. In order to give full play to the capabilities of the large model, the internal structure of the processing module 720 may be designed to be highly configurable and extensible so as to cope with various types of tasks and requirements in real scenarios.

[0127] In the example, after the AI ​​agent 700 obtains the first demand information, the processing module 720 can use the large model to process the first demand information of the object, obtain feedback information, and pass it to the output module 730.

[0128] It is understandable that although the large language model has excellent language understanding and generation capabilities, it is just like a human being and can only solve very limited tasks without any tools. When AI agent 700 is given the ability to call tools, it can perform tasks such as using a calculator to perform mathematical operations, using Python to perform data analysis, and using a search engine to perform weather forecasts.

[0129] In an example, the output module 730 may output the feedback information described above.

[0130] The AI ​​agent 700 according to the embodiment of the present disclosure can simply and effectively improve the level of intelligence, and improve flexibility and versatility.

[0131] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0132] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method as in the embodiment of the present disclosure.

[0133] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute a method as described in an embodiment of the present disclosure.

[0134] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the method according to the embodiment of the present disclosure is implemented.

[0135] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0136] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0137] A number of components in the device 800 are connected to an input / output (I / O) interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0138] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as an intelligent question-answering method based on a large model. For example, in some embodiments, the intelligent question-answering method based on a large model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the intelligent question-answering method based on the large model described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the large model-based intelligent question answering method in any other appropriate manner (eg, by means of firmware).

[0139] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0140] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0141] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0142] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0143] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0144] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0145] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0146] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An intelligent question answering method based on a large model, comprising: When it is determined that the target question is in a multimodal form, a non-text question in the target question is identified to obtain an identification result, wherein the target question also includes a text question; Rewriting the text question using the recognition result to obtain a rewritten text question; and The rewritten text question, the text question and the non-text question are input into a large model to obtain a target answer.

2. The method according to claim 1, wherein: The step of rewriting the text question using the recognition result to obtain a rewritten text question includes: Generate thought chain prompt information based on the text question and the recognition result, wherein the thought chain prompt information is used to instruct the question rewriting model to perform operations in a thought chain manner; and The thought chain prompt information is input into the problem rewriting model to obtain the rewriting text problem.

3. The method according to claim 1, wherein: The step of rewriting the text question using the recognition result to obtain a rewritten text question includes: Extracting keywords of preset types from the recognition results; and The pronouns in the text question are replaced with the keywords to obtain the rewritten text question.

4. The method according to any one of claims 1 to 3, wherein: The large model includes a multimodal large model and a fusion large model; The step of inputting the rewritten text question, the text question and the non-text question into the large model to obtain a target answer includes: Inputting the rewritten text question, the text question and the non-text question into the multimodal large model to obtain a plurality of initial answers; as well as Input a plurality of the initial answers into the fusion large model to obtain the target answer.

5. The method according to any one of claims 1 to 4, wherein: The identifying of the non-text questions in the target questions to obtain the identification results includes: Performing text element recognition and image element recognition on the non-text question respectively to obtain a plurality of initial recognition results; and Optimizing the plurality of initial recognition results to obtain the recognition result.

6. The method according to claim 5, wherein: The optimizing process of the plurality of the initial recognition results to obtain the recognition result comprises: In the case where the same content exists in a plurality of the initial recognition results, performing deduplication processing on the same content to obtain the recognition result; and When the plurality of initial recognition results do not have the same content, the recognition result is obtained based on the plurality of initial recognition results.

7. The method according to claim 1, further comprising: In the case where there is a reference relationship between the text question and the non-text question, it is determined that the text question and the non-text question are the target questions in a multimodal form, wherein the reference relationship between the text question and the non-text question is obtained through a reference control identifier of an interactive interface.

8. An intelligent question-answering device based on a large model, comprising: A recognition module, configured to recognize non-text questions in the target question when it is determined that the target question is in a multimodal form, and obtain a recognition result, wherein the target question also includes a text question; a rewriting module, configured to rewrite the text question using the recognition result to obtain a rewritten text question; and A processing module is used to input the rewritten text question, the text question and the non-text question into the big model to obtain a target answer.

9. The device according to claim 8, wherein: The rewriting module comprises: A generating submodule, used for generating thought chain prompt information based on the text question and the recognition result, wherein the thought chain prompt information is used for instructing the question rewriting model to perform operations in a thought chain manner; and The rewriting submodule is used to input the thought chain prompt information into the problem rewriting model to obtain the rewriting text problem.

10. The method according to claim 8, wherein: The rewriting module also includes: an extraction submodule, configured to extract keywords of a preset type from the recognition result; and The replacement submodule is used to replace the pronouns in the text question with the keywords to obtain the rewritten text question.

11. The device according to any one of claims 8 to 10, wherein: The large model includes a multimodal large model and a fusion large model; The processing module comprises: a processing submodule, for inputting the rewritten text question, the text question and the non-text question into the multimodal macro model to obtain a plurality of initial answers; and The fusion submodule is used to input the multiple initial answers into the fusion model to obtain the target answer.

12. The device according to any one of claims 8 to 11, wherein: The identification module comprises: A recognition submodule, configured to perform text element recognition and image element recognition on the non-text question to obtain a plurality of initial recognition results; and The optimization submodule is used to optimize the multiple initial recognition results to obtain the recognition result.

13. The device according to claim 12, wherein: The optimization submodule comprises: a deduplication unit, configured to, when the same content exists in a plurality of the initial recognition results, perform deduplication processing on the same content to obtain the recognition result; and The determination unit is used to obtain the recognition result based on the multiple initial recognition results when the multiple initial recognition results do not have the same content.

14. The apparatus according to claim 8, further comprising: A determination module is used to determine that the text question and the non-text question are the target questions in a multimodal form when there is a reference relationship between the text question and the non-text question, wherein the reference relationship between the text question and the non-text question is obtained through a reference control identifier of an interactive interface.

15. An artificial intelligence agent, comprising: An input module, used for receiving input information; a processing module, configured to determine a target task based on the input information received by the input module, determine a large language model based on the target task, and execute the method according to any one of claims 1 to 7 by calling the large language model; An output module is used to output the output information obtained by the processing module.

16. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

17. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

18. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images