Intelligent question and answer method, device and equipment based on large model, intelligent agent and medium

Abstract

The present disclosure provides a large model-based intelligent question answering method, relates to the technical field of artificial intelligence, and particularly relates to the technical fields of computer vision, deep learning, large models, and the like. The specific implementation scheme is as follows: in the case of determining that a target question is in a multi-modal form, a non-text question in the target question is identified to obtain an identification result, wherein the target question also includes a text question; the identification result is used to rewrite the text question to obtain a rewritten text question; and the rewritten text question, the text question, and the non-text question are input into a large model to obtain a target answer. The present disclosure provides a large model-based intelligent question answering device, an electronic device, an intelligent agent, a medium, and a product.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of computer vision, deep learning, and large-scale models. Specifically, it relates to an intelligent question-answering method, device, electronic device, intelligent agent, storage medium, and program product based on a large-scale model. Background Technology

[0002] With the rapid advancement of deep learning technology, large-scale model products have gradually become the mainstay of the intelligent dialogue service field. Trained on massive amounts of data, large-scale models can capture complex language patterns and user intent, providing users with personalized and intelligent dialogue experiences. However, how to enable large-scale models to accurately understand the user's true intent has become a key research focus. Summary of the Invention

[0003] This disclosure provides a method, apparatus, electronic device, storage medium, and program product for intelligent question answering based on a large model.

[0004] According to one aspect of this disclosure, a large-model-based intelligent question answering method is provided, comprising: when the target question is determined to be in a multimodal form, identifying non-textual questions in the target question to obtain identification results, wherein the target question also includes textual questions; rewriting the textual questions using the identification results to obtain rewritten textual questions; and inputting the rewritten textual questions, textual questions and non-textual questions into a large model to obtain the target answer.

[0005] According to another aspect of this disclosure, a large-model-based intelligent question-answering device is provided, comprising: an identification module, configured to identify non-textual questions in the target question when the target question is determined to be in a multimodal form, and obtain an identification result, wherein the target question also includes textual questions; a rewriting module, configured to rewrite the textual questions using the identification result, and obtain a rewritten textual question; and a processing module, configured to input the rewritten textual question, the textual question, and the non-textual question into the large-model to obtain the target answer.

[0006] According to another aspect of this disclosure, an artificial intelligence agent is provided, comprising: an input module for receiving input information; a processing module for determining a target task based on the input information received by the input module, determining a large language model based on the target task, and executing the above-described method by invoking the large language model; and an output module for outputting the output information obtained by the processing module.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the methods as disclosed herein.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods as disclosed herein.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method as disclosed herein.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0012] Figure 1 The illustration schematically shows an exemplary system architecture for applying a large-model-based intelligent question-answering method and apparatus according to embodiments of the present disclosure;

[0013] Figure 2 A flowchart illustrating a large-model-based intelligent question-answering method according to an embodiment of the present disclosure is shown schematically.

[0014] Figure 3 The illustration shows the effect of determining a multimodal objective problem according to an embodiment of the present disclosure;

[0015] Figure 4 A flowchart illustrating the determination of identification results according to an embodiment of the present disclosure is shown schematically;

[0016] Figure 5 A flowchart illustrating a large-model-based intelligent question-answering method according to another embodiment of the present disclosure is shown.

[0017] Figure 6 A block diagram of a large-model-based intelligent question-answering device according to an embodiment of the present disclosure is shown schematically.

[0018] Figure 7 A schematic diagram illustrating the structure of an intelligent agent of artificial intelligence according to embodiments of the present disclosure; and

[0019] Figure 8A block diagram of an electronic device suitable for implementing a large-model-based intelligent question-answering method according to embodiments of the present disclosure is illustrated. Detailed Implementation

[0020] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0021] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of users' personal information comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals.

[0022] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0023] Figure 1 The illustration schematically depicts an exemplary system architecture for applying large-model-based intelligent question-answering methods and apparatus according to embodiments of the present disclosure.

[0024] It is important to note that Figure 1 The examples shown are merely examples of system architectures that can be applied to embodiments of this disclosure, to help those skilled in the art understand the technical content of this disclosure, but do not imply that embodiments of this disclosure cannot be used in other devices, systems, environments, or scenarios. For example, in another embodiment, an exemplary system architecture for applying the intelligent question-answering method and apparatus based on large models may include a terminal device, but the terminal device can implement the intelligent question-answering method and apparatus based on large models provided in the embodiments of this disclosure without interacting with the server.

[0025] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0026] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).

[0027] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0028] Server 105 can be a server that provides various services, such as a backend management server that supports the content browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0029] It should be noted that the intelligent question-answering method based on a large model provided in this disclosure can generally be executed by terminal devices 101, 102, or 103. Accordingly, the intelligent question-answering device based on a large model provided in this disclosure can also be disposed in terminal devices 101, 102, or 103.

[0030] Alternatively, the large-model-based intelligent question-answering method provided in this embodiment can generally be executed by server 105. Correspondingly, the large-model-based intelligent question-answering device provided in this embodiment can generally be located in server 105. The large-model-based intelligent question-answering method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the large-model-based intelligent question-answering device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0031] For example, when a user enters a target question on the interactive interface, terminal devices 101, 102, and 103 can acquire the target question and then send it to server 105. Server 105 then identifies and rewrites the target question and outputs the target answer. Alternatively, a server or server cluster capable of communicating with terminal devices 101, 102, 103, and / or server 105 can identify and rewrite the target question and output the target answer.

[0032] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0033] It should be noted that the sequence numbers of the operations in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0034] In some large model-based question-answering scenarios, users may combine text modalities with other modalities to ask questions to the large model. The input text may be relatively brief, making it difficult for the large model to understand the user's true intentions and correctly stimulate the model's knowledge, resulting in inaccurate question responses.

[0035] This disclosure provides an intelligent question-answering method based on a large model. Given that the target question is determined to be multimodal, the non-textual questions within the target question are identified to obtain identification results. The target question also includes textual questions. The identification results are used to rewrite the textual questions to obtain rewritten textual questions. The rewritten textual questions, the textual questions, and the non-textual questions are then input into a large model to obtain the target answer.

[0036] When the target question is multimodal, non-textual questions are identified, and the identification results are used to rewrite the textual question, resulting in a more comprehensive and accurate rewritten textual question that better aligns with the user's intent. Subsequently, the rewritten textual question, the original textual question, and the non-textual question are input into a large model for processing, promoting the effective integration and complementarity of textual, non-textual, and rewritten textual questions, thereby improving the accuracy and relevance of the answers to the question.

[0037] Figure 2 A flowchart illustrating a large-model-based intelligent question-answering method according to an embodiment of the present disclosure is shown.

[0038] like Figure 2 As shown, the method includes operations S210~S230.

[0039] In operation S210, if the target problem is determined to be in a multimodal form, the non-textual problems in the target problem are identified to obtain the identification results. The target problem also includes textual problems.

[0040] In operation S220, the recognition results are used to rewrite the text question, resulting in the rewritten text question.

[0041] In operation S230, the rewritten text question, text question and non-text question are input into the large model to obtain the target answer.

[0042] In embodiments of this disclosure, the multimodal form may include a textual modal form and a non-textual modal form. Correspondingly, the target question may consist of a textual question and a non-textual question. The non-textual question represents information or data in a non-textual modal form. The textual question may be a question in a textual modal form. As a component of the target question, the textual question is typically a concise, broad, or vague question.

[0043] Non-textual modalities can include images, audio, video, and other modalities. The target problem in a multimodal form can be a combination of a text problem and at least one of the modalities of images, audio, video, etc. For example, the target problem in a multimodal form can be a combination of a text problem and one or more images, a combination of a text problem and video, a combination of a text problem and audio, a combination of a text problem and audio, and so on.

[0044] In some embodiments, textual and non-textual questions may be related in content. Textual questions may provide background, explanation, or guidance for non-textual questions; or, non-textual questions may provide background, explanation, or guidance for textual questions.

[0045] In some embodiments, textual and non-textual questions can complement each other to form a complete description of the problem. For example, a textual question might be "What song can this play?", while the corresponding non-textual question might be a picture of the musical instrument.

[0046] Once the target problem is determined to be multimodal, the types of non-textual modalities can be identified first. Then, a matching method can be selected to identify the non-textual problem based on its type. For example, image recognition technology can be used to identify images, speech recognition technology can be used to identify audio, and video analysis technology can be used to identify video.

[0047] The recognition results characterize key features of non-textual problems. For example, if the non-textual modality is an image, the recognition result could be objects, scenes, or faces in the image. If the non-textual modality is audio, the recognition result could be the speech content, speaker identity, or emotion. If the non-textual modality is video, the recognition result could be key information, actions, events, objects, and relationships between them in the video.

[0048] Rewriting text questions using recognition results can be achieved by combining the recognition results with the text question, resulting in a more complete and specific rewritten text question. This makes the rewritten text question more aligned with the user's intent.

[0049] In some embodiments, the recognition result is combined with the text question to rewrite the text question. This can be achieved by reconstructing the text question using the recognition result to obtain the rewritten text question. This disclosure is not limited to this; the recognition result can also be added to the text question as supplementary information, a scenario, a condition, or a constraint to obtain the rewritten text question. Alternatively, the original text question can be decomposed into multiple sub-questions, each of which is associated with a certain aspect of the recognition result, and the rewritten text question can be obtained based on these multiple sub-questions.

[0050] After rewriting the text question to obtain the rewritten text question, the rewritten text question, the text question, and the non-text question are input into the large model. The large model can retrieve relevant knowledge based on the rewritten text question, the text question, and the non-text question, and obtain the target answer based on the retrieved relevant knowledge.

[0051] A large model can refer to a deep learning model with a massive number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. The large model involved in this disclosure can be a general-purpose large model, or it can be an expert large model obtained through fine-tuning based on specific needs; this disclosure does not limit the scope of the large model. Large models can be used to process information of any modality, such as images, text, video, and audio.

[0052] According to embodiments of this disclosure, when the target question is in a multimodal form, by identifying non-textual questions and using the identification results to rewrite the textual question, a more comprehensive and accurate rewritten textual question is obtained, making the rewritten textual question more in line with the user's intent. Inputting the rewritten textual question, the original textual question, and the non-textual question into a large model for processing promotes the effective integration and complementarity of textual questions, non-textual questions, and rewritten textual questions, thereby improving the accuracy and relevance of the question answer.

[0053] According to embodiments of this disclosure, when a referencing relationship exists between textual and non-textual questions, the textual and non-textual questions are determined to be target questions in a multimodal form. The referencing relationship between textual and non-textual questions is obtained through a referencing control in the interactive interface.

[0054] Reference controls can be elements or mechanisms on an interactive interface used to clarify the reference relationship between textual and non-textual questions. Reference controls can indicate the association between two things through labels, hyperlinks, or embedded objects.

[0055] In some embodiments, in response to user actions on interactive information within the interface, a reference control can be displayed at the location of the interactive information. The interactive information may include text questions, non-text questions, and target answers. Users can reference the interactive information by clicking the reference control, which will create a reference control identifier at the input location of the current text question and also reference non-text content.

[0056] In some embodiments, text questions and non-text questions can be interactive information issued at different times within the interactive interface. When a user enters a text question, they can reference a non-text question using a reference control, and the current input position will display the text question, the non-text question, and the target question identified by the reference control.

[0057] Figure 3 The illustration shows the effect of determining a multimodal objective problem according to an embodiment of the present disclosure.

[0058] like Figure 3 As shown, the user first inputs a non-text question 301 containing an image. After several rounds of interaction, when the user inputs a text question 302 in the second instance, the non-text question 301 can be referenced again through a reference control. Below the text question 302, a reference control identifier 303 and the referenced non-text question 301 are displayed. By identifying the reference control identifier 303, it can be determined that there is a reference relationship between the text question 302 and the non-text question 301, thus determining that the target question input this time is in a multimodal form.

[0059] In some embodiments, the target question can also be determined to be in a multimodal form based on the input time and positional relationship between the text question and the non-text question, or by reference information to the non-text question contained in the text question. For example, if a user inputs both a text question and a non-text question simultaneously, the current input can be determined to be in a multimodal form. Alternatively, if the text question and the non-text question are in adjacent positions, a relationship between them can be determined, thus identifying the current input as a multimodal target question. Furthermore, if the text question contains explicit reference information, such as "above image" or "previous image," the current input can be identified as a multimodal target question.

[0060] According to embodiments of this disclosure, by using reference control identifiers to determine the reference relationship between text questions and non-text questions, associated text questions and non-text questions can be quickly located, maintaining the continuity between interactive information and improving the accuracy and efficiency of interactive information processing.

[0061] According to embodiments of this disclosure, when the target problem is determined to be in a multimodal form, the non-textual aspects of the target problem are identified to obtain identification results. This identification process may include: performing text element identification and image element identification on the non-textual aspects separately to obtain multiple initial identification results. The multiple initial identification results are then optimized to obtain the final identification result.

[0062] In embodiments of this disclosure, non-textual modal forms may be modal forms such as images and videos.

[0063] Text element recognition for non-text problems can involve identifying text or numbers in an image. Image element recognition for non-text problems can involve identifying specific objects in an image, as well as the objects' actions, shapes, colors, and spatial relationships.

[0064] Multiple visual models can be used to identify various text and image elements in non-text problems. For example, a text recognition model can identify text elements in non-text problems, a plant recognition model can identify plant elements, an animal recognition model can identify animal elements, and a subject recognition model can identify subject elements other than animal or plant elements in non-text problems. By using multiple visual models in parallel processing, both recognition accuracy and processing efficiency can be improved.

[0065] Each identified text element or image element can be used as an initial recognition result, and multiple recognition results can be obtained based on the identified text and image elements.

[0066] In some embodiments, optimizing multiple initial recognition results to obtain a recognition result may include correcting multiple initial recognition results. For example, text recognition results may be corrected to correct grammatical errors or spelling errors.

[0067] In some embodiments, optimizing multiple initial recognition results to obtain a recognition result may include filtering multiple initial recognition results to remove noise information.

[0068] In some embodiments, optimizing multiple initial recognition results to obtain a recognition result may include fusing multiple initial recognition results. The resulting recognition result may be a description of a non-textual problem determined based on text elements and image elements.

[0069] According to embodiments of this disclosure, by simultaneously recognizing text elements and image elements in non-text problems, information in non-text problems can be comprehensively captured. Optimizing multiple initial recognition results can further reduce recognition errors and noise, and improve recognition accuracy.

[0070] According to embodiments of this disclosure, optimizing multiple initial recognition results to obtain a recognition result may further include: if multiple initial recognition results contain identical content, performing deduplication on the identical content to obtain a recognition result; or, if multiple initial recognition results do not contain identical content, obtaining a recognition result based on the multiple initial recognition results.

[0071] The presence of identical content in multiple initial recognition results may be due to several reasons, including the existence of repeated elements in non-textual problems. For example, a certain text or symbol may appear multiple times in an image, leading to identical content in the initial recognition results. Alternatively, errors in the recognition algorithm may cause a single element to be identified as multiple identical elements, resulting in identical content in the initial recognition results. Another possibility is that multiple visual models were used in parallel processing of non-textual problems during the recognition process, causing a portion of an element to be processed by multiple visual models simultaneously, leading to identical content in the initial recognition results. Duplicate content leads to redundancy and confusion in the recognition results, thus requiring deduplication.

[0072] Multiple initial recognition results of the same category can be compared for similarity. When the similarity exceeds a preset threshold, it can be considered that there is duplicate content among the multiple initial recognition results. When it is determined that there is duplicate content among the multiple initial recognition results, the redundant duplicate content is deleted from the initial recognition results to obtain the recognition result.

[0073] If there is no identical content among the multiple initial recognition results, the multiple initial recognition results can be directly used as the recognition result, or the multiple initial recognition results can be spliced ​​or merged to obtain the recognition result.

[0074] According to embodiments of this disclosure, when multiple initial recognition results contain the same content, deduplication processing can eliminate duplicate recognition results, thereby avoiding redundancy and confusion and improving recognition accuracy.

[0075] Figure 4 A flowchart illustrating the determination of identification results according to an embodiment of the present disclosure is shown schematically.

[0076] like Figure 4 As shown, text element 402 and image element recognition are performed on the non-text problem 401 respectively. Image element recognition includes identifying animal elements 403, plant elements 404, and other main elements 405, resulting in multiple initial recognition results 406. These initial recognition results 406 are then optimized to obtain recognition result 407. The optimized recognition result 407 not only comprehensively captures information from the non-text problem, but also further reduces recognition errors and noise through optimization, thus improving recognition accuracy.

[0077] According to embodiments of this disclosure, rewriting a text question using the recognition results to obtain a rewritten text question may include: generating thought chain prompts based on the text question and the recognition results. The thought chain prompts instruct the large-scale question rewriting model to perform operations according to the thought chain. The thought chain prompts are then input into the large-scale question rewriting model to obtain the rewritten text question.

[0078] The chain-of-thinking approach represents a structured approach to thinking. It breaks down tasks into a series of interconnected and progressively deeper steps. Each step is a necessary link in the execution of the task, and each step is arranged in a specific logical order.

[0079] In some embodiments, after obtaining the recognition results for non-textual questions, the correlation between the recognition results and the textual questions can be analyzed first. For example, it can be understood how objects, scenes, or actions in the image are related to the textual questions, or how information in audio or video supplements or affects the understanding of the textual questions.

[0080] Based on the correlation between the recognition results and the text question, thought chain prompts can be generated. These prompts can be presented in natural language and may include specific tags or structures to instruct the large model to perform operations according to the thought chain.

[0081] For example, the text question is "What are the animals in the picture doing?" The non-text question is a picture of a cat chasing a mouse. The non-text question is identified as follows: the image contains a cat (object) and a mouse (object), and the cat is chasing the mouse (action). The text question is then understood to inquire about the behavior of the animals in the picture. The relationship between the identification result and the text question is analyzed to determine that the animals in the image are a cat and a mouse, and their behavior is chasing. Based on the relationship between the identification result and the text question, a thought chain prompt is generated, for example: first, identify the animals in the image; then, determine the behavioral relationship between them; finally, rewrite the text question based on this relationship. This thought chain prompt is input into the question rewriting model. Guided by the prompt, the question rewriting model generates a rewritten text question, such as "What is the cat in the picture chasing?" or "What action is the cat in the picture taking?".

[0082] According to embodiments of this disclosure, by generating thought chain prompts based on the recognition results and text questions, the problem rewriting model can be guided to rewrite the text questions in a logically clear and step-by-step manner, thereby achieving accurate rewriting of the text questions and improving the accuracy of target problem processing.

[0083] According to embodiments of this disclosure, rewriting a text question using the recognition results to obtain a rewritten text question may include: extracting keywords of a preset type from the recognition results; and replacing pronouns in the text question with the keywords to obtain the rewritten text question.

[0084] The pre-defined keyword type is not the key subject in the text question; it can be keywords such as names of people, places, times, and events. The text question may contain pronouns such as "he," "she," "it," "here," and "there," whose specific referents are not clear enough, causing large models to potentially fail to accurately identify the user's intent. By replacing the keywords extracted from the recognition results with the corresponding pronoun positions in the text question, a more specific and clear rewritten text question is generated.

[0085] For example, the text question is "What song can this play?" and the non-text question is a picture of a ukulele. By recognizing the picture, the result is "ukulele". Then, by replacing "this" in the text question with "ukulele", the rewritten text question becomes "What song can a ukulele play?".

[0086] According to embodiments of this disclosure, by replacing relevant pronouns with keywords of a preset type, the rewritten text question can be made more specific and clear, thereby facilitating subsequent answers or processing.

[0087] According to embodiments of this disclosure, the large model includes a multimodal large model and a fused large model. Inputting rewritten text questions, text questions, and non-text questions into the large model to obtain the target answer may include: inputting the rewritten text questions, text questions, and non-text questions into the multimodal large model to obtain multiple initial answers; and inputting the multiple initial answers into the fused large model to obtain the target answer.

[0088] Multimodal large models are capable of understanding and processing information from different modalities. They can internally fuse information from different modalities; for example, when both text and non-text questions are input simultaneously, the model can combine the text and non-text questions to generate an answer.

[0089] The fusion model can receive multiple initial answers from the multimodal model and, through further analysis and reasoning, integrate the useful information from the initial answers, remove redundant or erroneous parts, thereby generating a more refined and accurate final answer.

[0090] In some embodiments, a training dataset of a multimodal large model, including text questions and related images, videos, and audio, can be collected to train an initial multimodal large model, resulting in a multimodal large model. The initial answers generated by the initial multimodal large model can be used as the training dataset for a fusion large model to train the initial fusion large model, thus obtaining the fusion large model.

[0091] In some embodiments, inputting the rewritten text question, the text question, and the non-text question into a multimodal large model to obtain multiple initial answers may include: inputting the rewritten text question, the text question, and the non-text question into the multimodal large model, processing the text question and the non-text question using the multimodal large model to obtain a first initial answer; processing the rewritten text question and the non-text question together using the multimodal large model to obtain a second initial answer; and inputting the first initial answer and the second initial answer into a fusion large model to obtain the target answer.

[0092] According to embodiments of this disclosure, since the rewritten text question may also contain inaccuracies, the initial answer obtained based on the rewritten text question may be inaccurate. By inputting the text question, non-text question, and rewritten text question into a multimodal large model to obtain multiple initial answers, and then using a fusion large model to fuse the multiple initial answers, inaccurate initial answers can be corrected, thereby obtaining a target answer that is closer to the truth.

[0093] Figure 5 A flowchart illustrating a large-model-based intelligent question-answering method according to another embodiment of this disclosure is shown.

[0094] like Figure 5 As described above, when the target question is determined to be in a multimodal form, the non-text question 502 within the target question is identified, resulting in an identification result 503. Based on the identification result 503 and the text question 501, a thought chain hint 504 is generated. This hint 504 is input into the question rewriting model 505 to obtain a rewritten text question 506. The rewritten text question 506 is not only more comprehensive and accurate but also better reflects the user's intent. The text question 501, the non-text question 502, and the rewritten text question 506 are input into the multimodal model 507 to obtain multiple initial identification results. These initial identification results are then input into the fusion model 508 to obtain the target answer 509. The resulting target answer 509 integrates all valid information from the text question 501, the non-text question 502, and the rewritten text question 506, improving the accuracy and relevance of the question answer.

[0095] Figure 6 A block diagram of a large-model-based intelligent question-answering device according to an embodiment of the present disclosure is shown schematically.

[0096] like Figure 6 As shown, the device includes an identification module 610, a rewriting module 620, and a processing module 630.

[0097] The recognition module 610 is used to identify non-textual questions within the target question when it is determined that the target question is in a multimodal form, and to obtain the recognition result. The target question also includes textual questions.

[0098] The rewriting module 620 is used to rewrite the text question using the recognition results to obtain the rewritten text question.

[0099] The processing module 630 is used to input rewritten text questions, text questions, and non-text questions into the large model to obtain the target answer.

[0100] According to embodiments of this disclosure, the rewriting module 620 includes a generation submodule and a rewriting submodule.

[0101] The generation submodule is used to generate thought chain prompts based on the text question and recognition results. The thought chain prompts are used to instruct the large model to perform operations in a thought chain manner.

[0102] The rewriting submodule is used to input the thought chain hints into the problem rewriting model to obtain the rewritten text problem.

[0103] According to embodiments of this disclosure, the rewrite module 620 further includes an extraction submodule and a replacement submodule.

[0104] The extraction submodule is used to extract keywords of a preset type from the recognition results; and

[0105] The Replace submodule is used to replace pronouns in a text problem with keywords, resulting in a rewritten text problem.

[0106] According to embodiments of this disclosure, the large model includes a multimodal large model and a fused large model. The processing module 630 includes a processing submodule and a fusion submodule.

[0107] The processing submodule is used to input rewritten text problems, text problems, and non-text problems into the multimodal large model to obtain multiple initial answers.

[0108] The fusion submodule is used to input multiple initial answers into the fusion model to obtain the target answer.

[0109] According to embodiments of this disclosure, the identification module 610 includes an identification submodule and an optimization submodule.

[0110] The recognition submodule is used to perform text element recognition and image element recognition separately for non-text problems, obtaining multiple initial recognition results; and

[0111] The optimization submodule is used to optimize multiple initial recognition results to obtain the final recognition result.

[0112] According to embodiments of this disclosure, the optimization submodule includes a deduplication unit and a determination unit.

[0113] The deduplication unit is used to remove duplicate content from multiple initial recognition results, thus obtaining the final recognition result; and

[0114] The determining unit is used to obtain the identification result based on multiple initial identification results when there are no identical contents in multiple initial identification results.

[0115] According to embodiments of this disclosure, the apparatus further includes a determining module.

[0116] The determination module is used to identify whether the text problem and the non-text problem are multimodal target problems when a referencing relationship exists between them. The referencing relationship between the text problem and the non-text problem is obtained through the referencing control in the interactive interface.

[0117] According to embodiments of this disclosure, any plurality of modules among the identification module 610, rewriting module 620, and processing module 630 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the identification module 610, rewriting module 620, and processing module 630 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the identification module 610, rewriting module 620, and processing module 630 may be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0118] Figure 7 A schematic block diagram of an artificial intelligence agent according to an embodiment of the present disclosure is shown.

[0119] In embodiments of this disclosure, such as Figure 7 As shown, the AI ​​agent 700 may include an input module 710, a processing module 720, and an output module 730.

[0120] Input module 710 is used to receive input information;

[0121] The processing module 720 is used to determine the target task based on the input information received by the input module, determine the large model based on the target task, and execute the intelligent question answering method based on the large model according to the embodiments of this disclosure by calling the large model.

[0122] Output module 730 is used to output the output information obtained by the processing module.

[0123] According to embodiments of this disclosure, the input module 710 is responsible for receiving or sensing information such as queries, requests, instructions, signals, or data from the outside world (e.g., users or the external environment), and converting it into a format that the AI ​​agent 700 can understand and process. The input module 710 is the primary link for the AI ​​agent 800 to interact with the outside world, enabling the AI ​​agent 700 to efficiently and accurately obtain necessary "sensory" information from the outside world and respond to this information.

[0124] In the example, input module 710 can input the target problem, rewrite text problem, etc., as described above.

[0125] In the example, processing module 720 is the core support for the AI ​​agent 700's ability to handle complex tasks. Processing module 720 can execute the intelligent question-answering method based on a large model described above.

[0126] In the example, the performance of the processing module 720 is closely related to the large model on which the AI ​​agent 700 is based. To fully leverage the capabilities of the large model, the internal structure of the processing module 720 can be designed to be highly configurable and scalable to handle various types of tasks and requirements in real-world scenarios.

[0127] In the example, after the AI ​​agent 700 obtains the first demand information, the processing module 720 can use a large model to process the object's first demand information, obtain feedback information, and pass it to the output module 730.

[0128] Understandably, while large language models possess excellent language understanding and generation capabilities, like humans, their ability to solve tasks is limited without the aid of any tools. Once the AI ​​agent 700 is given the ability to invoke tools, it can perform tasks such as using a calculator to complete mathematical calculations, using Python to perform data analysis, and using a search engine to create weather forecasts.

[0129] In the example, output module 730 can output the feedback information described above.

[0130] The AI ​​agent 700 according to embodiments of this disclosure can simply and effectively improve the level of intelligence, as well as enhance flexibility and versatility.

[0131] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0132] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method as described in the embodiments of the present disclosure.

[0133] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause a computer to perform methods as described in embodiments of the present disclosure.

[0134] According to embodiments of the present disclosure, a computer program product includes a computer program that, when executed by a processor, implements the methods as described in embodiments of the present disclosure.

[0135] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0136] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0137] Multiple components in device 800 are connected to input / output (I / O) interface 805, including: input unit 806, such as a keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as a disk, optical disk, etc.; and communication unit 809, such as a network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0138] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as a large-model-based intelligent question answering method. For example, in some embodiments, the large-model-based intelligent question answering method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the large-model-based intelligent question answering method described above can be performed. Alternatively, in other embodiments, computing unit 801 may be configured to perform a large-model-based intelligent question-answering method by any other suitable means (e.g., by means of firmware).

[0139] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0140] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0141] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0142] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0143] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0144] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0145] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0146] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An intelligent question-answering method based on a large model, wherein the large model includes a multimodal large model and a fusion large model, the method comprising: When the target problem is determined to be in a multimodal form, the non-textual questions in the target problem are identified to obtain the identification results, wherein the target problem also includes textual questions; The text question is rewritten using the recognition results to obtain the rewritten text question; The rewritten text problem, the text problem, and the non-text problem are input into the multimodal large model. The multimodal large model is used to process the text problem and the non-text problem to obtain a first initial answer. The multimodal large model is then used to process the rewritten text problem and the non-text problem in combination to obtain a second initial answer. The first initial answer and the second initial answer are input into the fusion model to obtain the target answer; and When there is a referencing relationship between the text question and the non-text question, the text question and the non-text question are determined to be the target question in a multimodal form, wherein the text question and the non-text question are interactive information issued at different times in the interactive interface. The non-text question is referenced again by a reference control, and the referencing relationship between the text question and the non-text question is identified by displaying the reference control icon and the referenced non-text question on the interactive interface.

2. The method according to claim 1, wherein, The step of rewriting the text question using the recognition result to obtain the rewritten text question includes: Based on the text question and the recognition result, a thought chain prompt is generated, wherein the thought chain prompt is used to instruct the large-scale question rewriting model to perform operations according to the thought chain approach; and The thought chain prompts are input into the problem rewriting model to obtain the rewritten text problem.

3. The method according to claim 1, wherein, The step of rewriting the text question using the recognition result to obtain the rewritten text question includes: Extract keywords of a preset type from the recognition results; and The rewritten text question is obtained by replacing pronouns in the text question with the keywords.

4. The method according to any one of claims 1 to 3, wherein, The process of identifying non-textual issues within the target problem to obtain identification results includes: For the non-text problem, text element recognition and image element recognition were performed respectively, resulting in multiple initial recognition results; and The initial recognition results are optimized to obtain the final recognition result.

5. The method according to claim 4, wherein, The optimization process of multiple initial recognition results to obtain the recognition result includes: If multiple initial recognition results contain the same content, the duplicate content is removed to obtain the final recognition result; and If there are no identical contents in the multiple initial identification results, the identification result is obtained based on the multiple initial identification results.

6. A smart question-answering device based on a large model, wherein the large model includes a multimodal large model and a fusion large model, the device comprising: The recognition module is used to identify non-textual questions in the target question when the target question is determined to be in a multimodal form, and to obtain a recognition result, wherein the target question also includes textual questions; The rewriting module is used to rewrite the text question using the recognition results to obtain the rewritten text question; The processing submodule inputs the rewritten text problem, the text problem, and the non-text problem into the multimodal large model, processes the text problem and the non-text problem using the multimodal large model to obtain a first initial answer, and processes the rewritten text problem and the non-text problem together using the multimodal large model to obtain a second initial answer. The fusion submodule inputs the first initial answer and the second initial answer into the fusion model to obtain the target answer; and The determination module, when there is a referencing relationship between the text question and the non-text question, determines that the text question and the non-text question are the target question in a multimodal form, wherein the text question and the non-text question are interactive information issued at different times in the interactive interface, and the non-text question is referenced again by a reference control, and the referencing relationship between the text question and the non-text question is identified by displaying the reference control icon and the referenced non-text question on the interactive interface.

7. The apparatus according to claim 6, wherein, The rewriting module includes: A generation submodule is used to generate thought chain prompts based on the text question and the recognition result, wherein the thought chain prompts instruct the large-scale question rewriting model to perform operations according to the thought chain approach; and The rewriting submodule is used to input the thought chain prompt information into the problem rewriting model to obtain the rewritten text problem.

8. The apparatus according to claim 6, wherein, The rewriting module also includes: An extraction submodule is used to extract keywords of a preset type from the recognition results; and The replacement submodule is used to replace pronouns in the text question with the keywords to obtain the rewritten text question.

9. The apparatus according to any one of claims 6 to 8, wherein, The identification module includes: The recognition submodule is used to perform text element recognition and image element recognition on the non-text problem respectively, to obtain multiple initial recognition results; and The optimization submodule is used to optimize multiple initial recognition results to obtain the recognition result.

10. The apparatus according to claim 9, wherein, The optimization submodule includes: A deduplication unit is used to deduplicatize identical content in multiple initial recognition results to obtain the recognition result; and The determining unit is used to obtain the identification result based on the multiple initial identification results when there are no identical contents among the multiple initial identification results.

11. An intelligent agent of artificial intelligence, comprising: The input module is used to receive input information; The processing module is configured to determine a target task based on the input information received by the input module, determine a large language model based on the target task, and execute the method of any one of claims 1 to 5 by invoking the large language model; The output module is used to output the output information obtained by the processing module.

12. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 5.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 5.

14. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images