Answer generation method and device, equipment, storage medium and product

Abstract

The invention discloses an answer generation method and device, equipment, a storage medium and a product, relates to the technical field of artificial intelligence, and discloses a method for obtaining a target question corresponding to a dialogue instruction in response to the dialogue instruction; based on the target problem, a reference problem processing case matched with the target problem is searched from an experience knowledge base, the experience knowledge base comprises a plurality of problem processing cases, and the problem processing cases comprise the problem and a plurality of subtasks disassembled from the problem; splitting the target problem into a plurality of target sub-tasks based on the reference problem processing case through a large language model; and based on the plurality of target sub-tasks, generating an answer of the target question. The method can effectively improve the answer generation accuracy of the large language model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, storage medium and product for generating answers. Background Technology

[0002] In the field of artificial intelligence, large language models have emerged as a prominent force due to their powerful natural language processing capabilities. Based on a deep learning architecture, they build a deep understanding of language structure and semantics through unsupervised or supervised learning on massive amounts of text data, thereby enabling them to perform a variety of natural language tasks.

[0003] In related technologies, large language models can be used to generate answers to various questions, greatly facilitating people's rapid access to information. However, when faced with complex problems, large language models struggle to accurately grasp the key points and perform effective reasoning, resulting in low accuracy in the answers. Therefore, there is an urgent need to provide a method to improve the accuracy of answers generated by large language models.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this application is to provide a method, apparatus, device, storage medium, and product for generating answers, which can effectively improve the accuracy of answers generated by large language models.

[0006] To achieve the above objectives, this application proposes a method for generating answers, the method comprising:

[0007] In response to a dialogue command, obtain the target question corresponding to the dialogue command;

[0008] Based on the target problem, a reference problem-solving case matching the target problem is searched from the experience knowledge base. The experience knowledge base includes multiple problem-solving cases, and each problem-solving case includes a problem and multiple sub-tasks decomposed from the problem.

[0009] Using a large language model, based on the reference problem processing case, the target problem is decomposed into multiple target sub-tasks;

[0010] Based on the multiple target sub-tasks, an answer to the target question is generated.

[0011] Optionally, before searching for reference problem-solving cases matching the target problem from the experience knowledge base based on the target problem, the method further includes:

[0012] In response to a case addition command, obtain the question corresponding to the case addition command and multiple sub-tasks decomposed from the question;

[0013] Based on the case, add the corresponding problem to the instruction and the multiple sub-tasks decomposed from the problem to generate a problem handling case;

[0014] The generated problem-solving cases are stored in the experience knowledge base.

[0015] Optionally, the step of responding to a case addition instruction by obtaining the question corresponding to the case addition instruction and multiple subtasks decomposed from the question includes:

[0016] In response to the case addition command, retrieve the question edited in the case editing interface and the multiple subtasks decomposed from the question.

[0017] Optionally, generating an answer to the target question based on the plurality of target sub-tasks includes:

[0018] Based on the multiple target sub-tasks, multiple knowledge segments are retrieved from the multimodal knowledge base, and the knowledge segments contain image identifiers;

[0019] Using the large language model, and based on the multiple knowledge segments, an answer containing the image identifier is generated;

[0020] Replace the image identifier in the answer with the image corresponding to the image identifier to obtain the answer to the target question.

[0021] Optionally, before retrieving multiple knowledge segments from the multimodal knowledge base based on the multiple target sub-tasks, the method further includes:

[0022] In response to a knowledge addition instruction, retrieve the document corresponding to the knowledge addition instruction;

[0023] Replace the images in the document with the image identifiers corresponding to those images to obtain a plain text document.

[0024] The plain text document is divided into multiple knowledge segments, and the multiple knowledge segments are stored in the multimodal knowledge base.

[0025] Optionally, the method further includes:

[0026] The images are associated with their corresponding image identifiers and stored in the multimodal knowledge base;

[0027] Before replacing the image identifier in the answer with the image corresponding to the image identifier, the method further includes:

[0028] Retrieve the image corresponding to the image identifier from the multimodal knowledge base.

[0029] Optionally, generating an answer to the target question based on the plurality of target sub-tasks includes:

[0030] Retrieve knowledge metadata that matches the multiple target sub-tasks from a multimodal knowledge base;

[0031] Determine the knowledge segment to which the knowledge metadata belongs;

[0032] Using the large language model, the answer to the target question is generated based on the knowledge metadata and the knowledge paragraph to which the knowledge metadata belongs.

[0033] Optionally, the knowledge metadata includes at least one of the following metadata:

[0034] Basic metadata includes at least one of the following: knowledge type, knowledge source, applicable scenario, and update time of the knowledge segment to which it belongs;

[0035] Semantic metadata contains semantic information about the knowledge segment to which it belongs;

[0036] Related metadata, which includes the relationship between the knowledge paragraph and other knowledge paragraphs.

[0037] Optionally, generating an answer to the target question based on the plurality of target sub-tasks includes:

[0038] Based on the problem and the multiple target sub-tasks, the current problem-solving steps are generated using the large language model.

[0039] Perform the current problem-solving steps to obtain the problem-solving result;

[0040] Using the large language model, if the question processing result meets the answer conditions, the question processing result is used as the answer to the question; or, if the question processing result does not meet the answer conditions, the next question processing step is regenerated based on the question, the multiple subtasks, and the obtained question processing results, and the question processing result of the next question processing step is obtained, until the obtained question processing result meets the answer conditions.

[0041] Optionally, the current problem-handling step includes the execution tool to be invoked and the invocation parameters of the execution tool; the execution of the current problem-handling step to obtain the problem-handling result includes:

[0042] Based on the calling parameters, the execution tool is invoked to obtain the calling result;

[0043] Based on the invocation result, the large language model is used to generate the problem processing result for the current problem processing step.

[0044] Optionally, the execution tool includes at least one of the following:

[0045] Domain-specific large models, database query tools, business system interface calling tools, and search engines are available for different knowledge domains.

[0046] Optionally, the step of searching for reference problem-solving cases matching the target problem from the experience knowledge base based on the target problem includes:

[0047] The intent to identify the target question is performed to obtain the intent identification result;

[0048] If the target problem is determined to be a complex problem based on the intent recognition result, a reference problem-solving case matching the target problem is searched from the experience knowledge base based on the target problem.

[0049] Optionally, the method further includes:

[0050] If the target problem is determined to be a simple problem based on the intent recognition result, a target domain large model that matches the intent recognition result is selected from the domain model set, wherein the domain model set contains multiple domain large models for different knowledge domains;

[0051] The target domain model is used to generate the answer to the target question.

[0052] Optionally, generating the answer to the target question using the target domain big model includes:

[0053] Using the target domain big model, other domain big models in the domain model set besides the target domain big model are called to obtain answer reference information, and based on the answer reference information, the answer corresponding to the target question is generated.

[0054] Optionally, after generating the answer to the target question based on the plurality of target sub-tasks, the method further includes:

[0055] Obtain answer feedback information, which includes an answer quality score and / or content indicating deviations in the answer;

[0056] Based on the feedback information from the response, a knowledge base update operation is performed.

[0057] Optionally, the method further includes:

[0058] Obtain multiple processing samples of the large language model within a preset time period, the processing samples including the questions processed by the large language model and the answers corresponding to the questions;

[0059] Using the large language model, processing samples whose answer quality does not meet the quality criteria are selected from the multiple processing samples;

[0060] Based on the selected processing samples, a knowledge base update operation is performed.

[0061] Optionally, before searching for reference problem-solving cases matching the target problem from the experience knowledge base based on the target problem, the method further includes:

[0062] The target problem is rewritten, and the rewriting process includes at least one of the following:

[0063] Based on historical dialogues, the content of the target question is supplemented;

[0064] Remove meaningless words from the target problem;

[0065] Replace the words in the target problem.

[0066] Furthermore, to achieve the above objectives, this application also proposes a response generation apparatus, the apparatus comprising:

[0067] The question acquisition module is used to acquire the target question corresponding to the dialogue command in response to the dialogue command;

[0068] The case search module is used to search for reference problem-solving cases that match the target problem from the experience knowledge base based on the target problem. The experience knowledge base includes multiple problem-solving cases, and each problem-solving case includes a problem and multiple sub-tasks decomposed from the problem.

[0069] The problem decomposition module is used to decompose the target problem into multiple target sub-tasks based on the reference problem processing case using a large language model;

[0070] The answer generation module is used to generate an answer to the target question based on the multiple target sub-tasks.

[0071] Optionally, the device further includes:

[0072] The case addition module is used to respond to a case addition command, obtain the problem corresponding to the case addition command and multiple sub-tasks decomposed from the problem; generate a problem handling case based on the problem corresponding to the case addition command and the multiple sub-tasks decomposed from the problem; and store the generated problem handling case in the experience knowledge base.

[0073] Optionally, the case adding module is used to respond to a case adding instruction to obtain the question edited in the case editing interface and multiple sub-tasks decomposed from the question.

[0074] Optionally, the answer generation module is configured to retrieve multiple knowledge segments from a multimodal knowledge base based on the multiple target sub-tasks, wherein the knowledge segments contain image identifiers; generate an answer containing the image identifiers based on the multiple knowledge segments using the large language model; and replace the image identifiers in the answer with the images corresponding to the image identifiers to obtain the answer corresponding to the target question.

[0075] Optionally, the device further includes:

[0076] The knowledge base construction module is used to respond to a knowledge addition instruction, obtain the document corresponding to the knowledge addition instruction; replace the images in the document with the image identifiers corresponding to the images to obtain a plain text document; divide the plain text document into multiple knowledge paragraphs, and store the multiple knowledge paragraphs in the multimodal knowledge base.

[0077] Optionally, the knowledge base construction module is further configured to associate the image with the corresponding image identifier and store it in the multimodal knowledge base;

[0078] The answer generation module is also used to query the image corresponding to the image identifier from the multimodal knowledge base.

[0079] Optionally, the answer generation module is configured to retrieve knowledge metadata matching the multiple target sub-tasks from a multimodal knowledge base; determine the knowledge paragraph to which the knowledge metadata belongs; and generate an answer corresponding to the target question based on the knowledge metadata and the knowledge paragraph to which the knowledge metadata belongs, using the large language model.

[0080] Optionally, the knowledge metadata includes at least one of the following metadata:

[0081] Basic metadata includes at least one of the following: knowledge type, knowledge source, applicable scenario, and update time of the knowledge segment to which it belongs;

[0082] Semantic metadata contains semantic information about the knowledge segment to which it belongs;

[0083] Related metadata, which includes the relationship between the knowledge paragraph and other knowledge paragraphs.

[0084] Optionally, the answer generation module is configured to generate a current question processing step based on the question and the multiple target sub-tasks using the large language model; execute the current question processing step to obtain a question processing result; and, if the question processing result satisfies the answer conditions using the large language model, use the question processing result as the answer to the question; or, if the question processing result does not satisfy the answer conditions, regenerate the next question processing step based on the question, the multiple sub-tasks, and the obtained question processing result, and obtain the question processing result of the next question processing step, until the obtained question processing result satisfies the answer conditions.

[0085] Optionally, the current problem-solving step includes the execution tool to be invoked and the invocation parameters of the execution tool;

[0086] The answer generation module is used to invoke the execution tool based on the invocation parameters to obtain the invocation result; and to generate the problem processing result of the current problem processing step based on the invocation result through the large language model.

[0087] Optionally, the execution tool includes at least one of the following:

[0088] Domain-specific large models, database query tools, business system interface calling tools, and search engines are available for different knowledge domains.

[0089] Optionally, the device further includes:

[0090] The intent recognition module is used to recognize the intent of the target question and obtain the intent recognition result;

[0091] The case search module is used to search for reference problem-solving cases that match the target problem from the experience knowledge base when the target problem is determined to be a complex problem based on the intent recognition result.

[0092] Optionally, the answer generation module is further configured to, when determining that the target question is a simple question based on the intent recognition result, select a target domain model from the domain model set that matches the intent recognition result, wherein the domain model set contains multiple domain models for different knowledge domains; and generate an answer corresponding to the target question through the target domain model.

[0093] Optionally, the answer generation module is used to obtain answer reference information by calling other domain large models in the domain model set other than the target domain large model through the target domain large model, and generate an answer corresponding to the target question based on the answer reference information.

[0094] Optionally, the device further includes:

[0095] The first update module is used to obtain answer feedback information, which includes the answer quality score and / or the content marked as having deviations in the answer; and to perform a knowledge base update operation based on the answer feedback information.

[0096] Optionally, the device further includes:

[0097] The second update module is used to obtain multiple processing samples of the large language model within a preset time period. The processing samples include the questions processed by the large language model and the answers corresponding to the questions. The large language model is used to filter out processing samples whose answer quality does not meet the quality conditions from the multiple processing samples. Based on the filtered processing samples, a knowledge base update operation is performed.

[0098] Optionally, the device further includes:

[0099] The problem rewriting module is used to rewrite the target problem, wherein the rewriting process includes at least one of the following:

[0100] Based on historical dialogues, the content of the target question is supplemented;

[0101] Remove meaningless words from the target problem;

[0102] Replace the words in the target problem.

[0103] In addition, to achieve the above objectives, this application also proposes an answer generation device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the answer generation method as described above.

[0104] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the answer generation method described above.

[0105] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the answer generation method described above.

[0106] One or more technical solutions proposed in this application have at least the following technical effects:

[0107] The answer generation scheme provided in this application, after obtaining the target question corresponding to the dialogue instruction, searches for reference question-handling cases matching the target question from an experience knowledge base. Since the question-handling cases in the experience knowledge base contain the question and multiple sub-tasks decomposed from the question, the reference question-handling cases matching the target question provide a referable model for handling the target question. Next, using a large language model, the target question is decomposed into multiple target sub-tasks based on the reference question-handling cases, and an answer to the target question is generated based on these multiple target sub-tasks. This approach effectively reduces the complexity of the question. Compared to having the large language model directly handle the complex overall question, decomposing it into multiple sub-tasks allows the large language model to focus more intently on analyzing and understanding each sub-task, thereby improving its ability to grasp the key points of the question. This results in a more relevant and logically sound answer. This scheme effectively improves the large language model's ability to handle complex questions and enhances the accuracy of the answers generated by the large language model. Attached Figure Description

[0108] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0109] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0110] Figure 1 This is a schematic diagram of an implementation environment for the response generation method of this application;

[0111] Figure 2 This is a flowchart illustrating the first embodiment of the response generation method for this application.

[0112] Figure 3 This is a schematic diagram of the newly added step in the second embodiment of the response generation method of this application;

[0113] Figure 4 A detailed step diagram of step S40 in the third embodiment of the answer generation method of this application;

[0114] Figure 5 A detailed step diagram of step S40 in the fourth embodiment of the answer generation method for this application;

[0115] Figure 6 This is a detailed step diagram of step S40 in the fifth embodiment of the answer generation method of this application;

[0116] Figure 7This is a schematic diagram of the newly added step in the sixth embodiment of the response generation method of this application;

[0117] Figure 8 This application provides a schematic diagram of a process for constructing a multimodal database.

[0118] Figure 9 A schematic diagram illustrating a problem-solving process provided in an embodiment of this application;

[0119] Figure 10 This application provides a schematic diagram illustrating a process for updating a database.

[0120] Figure 11 This is a schematic diagram of the module structure of the answer generation device in the embodiments of this application;

[0121] Figure 12 This is a schematic diagram of the device structure of the hardware operating environment involved in the answer generation method in this application embodiment.

[0122] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0123] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0124] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0125] Figure 1 This is a schematic diagram illustrating an implementation environment provided by an embodiment of this disclosure. See also... Figure 1 The implementation environment includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected via a wireless or wired network. For example, the terminal 101 is installed with a target application provided by the server 102, and the terminal 101 can perform functions such as data transmission and message interaction through the target application.

[0126] For example, terminal 101 is a computer, mobile phone, tablet computer, or other terminal. For example, the target application is a target application in the operating system of terminal 101, or a target application provided by a third party. For example, the target application is an information search application, a conversation application, a chat application, etc. For example, server 102 is the backend server corresponding to the target application. Accordingly, server 102 is a search application server, a conversation application server, a chat application server, etc.

[0127] In this application, terminal 101 is used to respond to an answer generation instruction, obtain the target question corresponding to the dialogue instruction, and send the target question to the server. Server 102 receives the target question, and based on the target question, searches for reference question handling cases that match the target question from the experience knowledge base. Then, using a large language model, based on the reference question handling cases, the target question is decomposed into multiple target sub-tasks, and an answer to the target question is generated based on the multiple target sub-tasks.

[0128] Alternatively, the above-mentioned answer generation process can also be completed by terminal 101 alone. Alternatively, terminal 101 can complete it through the installed target application. This application embodiment does not limit this.

[0129] The answer generation method provided in this application is applicable to various scenarios, such as intelligent customer service. In enterprise customer service, customers often raise various complex questions, such as questions about the combined use of product functions and after-sales service processes. When an intelligent customer service system applies this solution, after receiving a customer's question, it can find similar problem-solving cases from its experience knowledge base, break down complex questions into sub-tasks such as product function introductions and specific after-sales steps, and then use a large language model to generate accurate answers for each sub-task, quickly and efficiently resolving customer inquiries and improving customer satisfaction.

[0130] For example, in online education scenarios, students may raise complex subject-specific questions during their learning process, such as challenging math problems involving the integrated application of multiple knowledge points or historical problems requiring cross-event analysis. Online education platforms can utilize this approach by first acquiring the student's question, then referencing similar problem-solving cases in their knowledge base, and breaking down the complex problem into specific sub-tasks such as knowledge point analysis and logical reasoning steps. The large language model then generates detailed solutions based on these sub-tasks, helping students better understand and master the knowledge.

[0131] Another example is intelligent writing assistance. When users write on complex topics, such as academic papers or business plans, they may need to organize content from multiple aspects. Writing assistance tools, applying this approach, after understanding the user's writing needs (i.e., the target problem), find similar writing examples from an experience knowledge base. The writing task is then broken down into sub-tasks such as argument formulation, evidence collection, and logical structure building. Based on this, a large language model generates more logically sound content, assisting users in completing high-quality writing.

[0132] Figure 2 This is a flowchart illustrating the first embodiment of the method for generating responses to this application. (Refer to...) Figure 2 Taking the terminal as the executing entity as an example, the answer generation method includes the following steps S10 to S40:

[0133] Step S10: In response to the dialogue command, obtain the target question corresponding to the dialogue command.

[0134] A dialogue command refers to the instruction issued by a user when interacting with the question-and-answer system. This command can be text information entered by the user in the intelligent customer service interface, or voice content spoken to the voice assistant. It is the way the user expresses their needs, and the system will initiate subsequent processing based on this command. The target question is the problem that the user wants to solve, as determined by the dialogue command. For example, if a user mentions in the dialogue, "I want to know how to write a tender document," this is a target question, and the question-and-answer system needs to provide relevant answers around this question.

[0135] Optionally, after obtaining the target question, it may be rewritten, which involves a series of text processing operations to optimize its expression, making it easier for the system to understand and process, thereby improving the accuracy and quality of the answer. The rewriting process includes at least one of the following:

[0136] First, supplement the content of the target question based on historical dialogues. Historical dialogues refer to a series of interactive dialogues between the user and the system prior to the current dialogue. These records contain questions previously asked by the user and the system's answers.

[0137] For example, the historical question was "What are the product's functions?" The model provided information on 10 functions. The current target question is "Describe the third function in detail." By rewriting the target question, the resulting question is "Describe the ××× function in detail," where "×××" is the name of the third function extracted from the historical answers.

[0138] Second, remove meaningless words from the target question. Meaningless words refer to words in the target question that do not substantially help express the core intent of the question, such as some modal particles, auxiliary words, or redundant words unrelated to the topic of the question.

[0139] Third, replace words in the target question. That is, replace certain words in the target question with words that more accurately express the intent or are more suitable for information retrieval.

[0140] By supplementing the target question with information from historical dialogues, more relevant information can be obtained, enabling the question-and-answer system to gain a more comprehensive understanding of the user's intent and avoid misunderstandings caused by missing information. Removing meaningless words reduces interference, allowing the question-and-answer system to focus more on the key parts of the question and improve its ability to grasp the core of the problem. Replacing words makes the question clearer and more accurate, better matching the system's knowledge base and understanding methods, thus leading to a more accurate understanding of the user's question. The rewritten target question is easier for the system to process, guiding it to more accurately retrieve relevant information from the knowledge base and generate answers that better meet the user's needs, thereby improving the quality of the answers.

[0141] Step S20: Based on the target problem, search for reference problem-solving cases that match the target problem from the experience knowledge base. The experience knowledge base includes multiple problem-solving cases, which include the problem and multiple sub-tasks decomposed from the problem.

[0142] An experience knowledge base is a database storing a large amount of knowledge and information, including multiple problem-solving cases. These cases are summaries of experience accumulated from handling various problems in the past, serving as important references for the system to solve new problems. For example, problem-solving cases in the experience knowledge base can be manually labeled; that is, problems are manually defined and broken down into multiple sub-tasks, and the problem and these sub-tasks are then used to construct problem-solving cases. Alternatively, problem-solving cases in the experience knowledge base can be generated based on the model's historical data; that is, problem-solving cases are generated by combining problems previously processed by the model with the multiple sub-tasks obtained from breaking down the problems.

[0143] Problem-solving cases are the basic units in the experience knowledge base. Each case contains a problem and several sub-tasks broken down from that problem. Sub-tasks are smaller, easier-to-handle tasks derived from breaking down a complex problem. Decomposing complex problems into sub-tasks allows the system to analyze and process them more effectively, thereby improving problem-solving efficiency and accuracy. For example, for the problem "How to assemble a computer," sub-tasks might include "Selecting suitable hardware components," "Assembling hardware components," and "Connecting power lines." Similarly, for the problem "Developing a pricing plan for network security and data governance," sub-tasks might include "Retrieving information on network security and data governance products," "Solution design," "Breaking down the functional list of the solution and obtaining product specifications," "Obtaining pricing rules, functional prices, and bundled discounts," "Using the pricing system to generate a price list and checking its validity," and "Providing a complete pricing plan and price list."

[0144] Reference problem-solving cases are problem-solving cases that match the target problem and are searched from an experience knowledge base. For example, the similarity between the target problem and the problems in each problem-solving case in the experience knowledge base is determined, and the problem-solving case corresponding to the problem whose similarity to the target problem is greater than a preset threshold is selected as the reference problem-solving case. There can be one or more reference problem-solving cases; this application does not limit this.

[0145] Step S30: Using a large language model and based on reference problem-solving cases, the target problem is broken down into multiple target sub-tasks.

[0146] The target subtasks are specific subtasks obtained by the model after breaking down the target problem based on reference problem-solving cases. These subtasks are specifically determined for the current target problem and form the basis for generating answers to the target problem.

[0147] Step S40: Generate an answer to the target question based on multiple target sub-tasks.

[0148] For example, the target question and its multiple sub-tasks can be input into a large language model to obtain the model's answer to the target question. Alternatively, by rewriting the target question, the original target question, the rewritten target question, and the multiple sub-tasks can be input into the large language model to provide richer information to the model.

[0149] The answer generation scheme provided in this application, after obtaining the target question corresponding to the dialogue instruction, searches for reference question-handling cases matching the target question from an experience knowledge base. Since the question-handling cases in the experience knowledge base contain the question and multiple sub-tasks decomposed from the question, the reference question-handling cases matching the target question provide a referable model for handling the target question. Next, using a large language model, the target question is decomposed into multiple target sub-tasks based on the reference question-handling cases, and an answer to the target question is generated based on these multiple target sub-tasks. This approach effectively reduces the complexity of the question. Compared to having the large language model directly handle the complex overall question, decomposing it into multiple sub-tasks allows the large language model to focus more intently on analyzing and understanding each sub-task, thereby improving its ability to grasp the key points of the question. This results in a more relevant and logically sound answer. This scheme effectively improves the large language model's ability to handle complex questions and enhances the accuracy of the answers generated by the large language model.

[0150] Based on the first embodiment described above, a second embodiment of this application is proposed. Contents that are the same as or similar to the first embodiment can be referred to the above description and will not be repeated hereafter. (Refer to...) Figure 3 In the second embodiment, before step S10, steps S01 to S03 are included:

[0151] Step S01: In response to the case addition instruction, obtain the question corresponding to the case addition instruction and the multiple subtasks decomposed from the question.

[0152] The case addition command is an instruction issued by the system administrator to the system to add new problem-solving cases to the experience knowledge base. This command instructs the system to execute the relevant procedures for adding cases.

[0153] Optionally, in response to a case addition instruction, the system retrieves the question corresponding to the case addition instruction and multiple subtasks derived from the question, including: in response to a case addition instruction, the system retrieves the question being edited in the case editing interface and multiple subtasks derived from the question.

[0154] The case editing interface is an interactive interface for system administrators to input and edit information related to problem-solving cases. In this interface, system administrators can easily input the problem to be added to the experience knowledge base, as well as the multiple sub-tasks that the problem is broken down into. For example, the case editing interface includes auxiliary editing functions such as formatting, saving, and previewing to help users accurately and efficiently complete the case addition operation.

[0155] By providing a case editing interface, users can obtain manually edited questions and corresponding subtasks. On one hand, human experience can be used to identify suitable questions for inclusion in the knowledge base for reference. On the other hand, human expertise and experience can be leveraged to more accurately break down questions into subtasks, thus providing accurate references for solving similar problems in the future.

[0156] Step S02: Based on the case, add the problem corresponding to the instruction and the multiple sub-tasks decomposed from the problem to generate a problem handling case.

[0157] Step S03: Store the generated problem-solving cases in the experience knowledge base.

[0158] In this embodiment, by continuously responding to case addition instructions and storing new problem-solving cases in the experience knowledge base, the experience knowledge base can be continuously expanded and improved. As the number of cases increases, the experience knowledge base can cover a wider range of problem types and scenarios, providing richer reference for solving various target problems.

[0159] Based on the first embodiment described above, a third embodiment of this application is proposed. Contents that are the same as or similar to the first embodiment can be referred to the above description and will not be repeated hereafter. (Refer to...) Figure 4 In the third embodiment, step S40 includes steps S401 to S403:

[0160] Step S401: Based on multiple target sub-tasks, retrieve multiple knowledge segments from the multimodal knowledge base, where each knowledge segment contains an image identifier.

[0161] A multimodal knowledge base is a database that stores information in multiple modalities, including text, images, tables, and other data formats, providing a rich source of knowledge for answering questions.

[0162] A knowledge segment is a text fragment retrieved from a multimodal knowledge base. It contains knowledge content related to the target subtask and may include image identifiers to indicate image information associated with the knowledge segment.

[0163] An image identifier is a unique identifier for an image, which can contain symbols, codes, links, or descriptions of the image's content. It associates text information within a paragraph with a specific image. For example, this image identifier is in Markdown format, meaning its format is . The alternative text describes the image and is displayed when the image cannot be displayed.

[0164] Optionally, before retrieving multiple knowledge segments from the multimodal knowledge base based on multiple target sub-tasks, the terminal responds to the knowledge addition instruction by obtaining the document corresponding to the knowledge addition instruction; replaces the images in the document with the image identifiers corresponding to the images, obtaining a plain text document; divides the plain text document into multiple knowledge segments, and stores the multiple knowledge segments in the multimodal knowledge base.

[0165] A knowledge addition command is an operation command issued by a user or system to add knowledge to a multimodal knowledge base. It instructs the execution of subsequent knowledge addition processes, including retrieving the corresponding document and processing its content. A plain text document refers to a document in plain text format, where non-text elements such as images have been removed, retaining only the text information. Converting documents to plain text format facilitates easier storage, retrieval, and processing of the document content.

[0166] In this embodiment, images in the document are replaced with their corresponding image identifiers to obtain a plain text document. This plain text document is then stored in a multimodal knowledge base as multiple knowledge paragraphs. This approach allows for faster location of subtask-related content within the plain text when retrieving knowledge paragraphs from the multimodal knowledge base based on a target subtask. Compared to processing documents with complex formats such as images, the retrieval speed is faster, providing timely knowledge support for generating question answers and reducing response time. Furthermore, replacing images with image identifiers preserves the association between images and text while storing plain text knowledge paragraphs. This allows the multimodal knowledge base to effectively integrate text and image information. When generating question answers, images corresponding to the image identifiers can be accurately inserted into the answer as needed, achieving a rich combination of text and images, fully leveraging the advantages of multimodal information, and improving the quality and richness of the answer.

[0167] Step S402: Using a large language model, based on multiple knowledge paragraphs, generate an answer that includes image identifiers.

[0168] Step S403: Replace the image identifier in the answer with the image corresponding to the image identifier to obtain the answer to the target question.

[0169] Optionally, in addition to storing multiple knowledge segments in the multimodal knowledge base, the terminal also associates images with their corresponding image identifiers and stores them in the multimodal knowledge base. Accordingly, before replacing the image identifier in the answer with the corresponding image, the terminal queries the multimodal knowledge base for the image corresponding to the image identifier. This ensures that the image corresponding to the image identifier is quickly retrieved, thereby generating answers containing images more efficiently and improving the user experience.

[0170] In this embodiment, knowledge segments in the multimodal knowledge base may contain image identifiers. The large language model generates answers based on knowledge segments containing image identifiers, and the generated answers may also contain image identifiers. Subsequently, replacing the image identifiers in the model-generated answers with corresponding images yields a text-image hybrid answer. This solution enables large language models that do not natively support image processing to combine textual knowledge with image information to generate answers with images. This enhances the expressiveness and persuasiveness of the answers, helping users better understand the content.

[0171] Based on the first embodiment described above, a fourth embodiment of this application is proposed. Contents that are the same as or similar to the first embodiment can be referred to the above description and will not be repeated hereafter. (Refer to...) Figure 5 In the fourth embodiment, step S40 includes steps S404 to S406:

[0172] Step S404: Retrieve knowledge metadata that matches multiple target subtasks from the multimodal knowledge base.

[0173] Knowledge metadata is data about knowledge; it describes some characteristics and attributes of knowledge, such as the subject, source, creation time, and related keywords. This metadata helps the system to more accurately classify, retrieve, and manage knowledge, so that relevant knowledge segments can be quickly located and retrieved when needed.

[0174] Optionally, knowledge metadata includes at least one of the following metadata: basic metadata, containing at least one of the following: knowledge type, knowledge source, applicable scenario, and update time of the knowledge segment; semantic metadata, containing semantic information of the knowledge segment; and relational metadata, containing the relationship between the knowledge segment and other knowledge segments. In addition, relational metadata may also contain relationships between other data, such as header data in a table, or the management relationship of an image's location in a document.

[0175] Taking an enterprise intelligent assistant system as an example, the knowledge types in the basic metadata of knowledge segments in its multimodal knowledge base can include products, solutions, personnel information, enterprise policies and processes, and internal business systems. Applicable scenarios can include bidding, sales, and technical support. Semantic metadata can include key entities and content summaries in document segments, such as firewalls and data encryption.

[0176] It is understood that this application's embodiments differentiate various types of knowledge by annotating metadata for each knowledge segment, thus achieving a layered multimodal knowledge base. Taking the multimodal knowledge base of an enterprise intelligent assistant system as an example, the knowledge is divided into: a product layer, including product functions, technical parameters, operation manuals, qualification documents, etc.; a solution layer, including cross-product combination solutions, historical cases, and deployment guidelines; an enterprise personnel information layer, including information such as product line leaders, personnel responsibilities, technical support contacts, sales liaisons, and personal private knowledge; an enterprise system and process layer, including internal production, sales, human resources, finance, and R&D management systems and processes; and an internal business system layer, including user manuals and training courses for the enterprise's internal system Wiki (a collaborative hypertext system). This allows for efficient and accurate information retrieval using the layered knowledge base. Experiments show that the knowledge retrieval accuracy rate using the layered multimodal knowledge base of this application is over 95%, while the knowledge retrieval accuracy rate of traditional knowledge bases is only 70%.

[0177] In this embodiment, knowledge segments are comprehensively described using various types of metadata, enabling a more detailed characterization and management of knowledge. Basic metadata provides the fundamental attributes of knowledge, allowing users and the system to quickly understand the general situation of the knowledge; semantic metadata delves into the semantic level of knowledge, helping computers to better understand and process knowledge; and relational metadata establishes connections between knowledge, making the knowledge structure clearer and facilitating the integration and utilization of knowledge.

[0178] Different types of metadata provide multi-dimensional indexes for knowledge retrieval. Basic metadata can quickly filter relevant knowledge based on knowledge type, source, etc.; semantic metadata can perform precise retrieval based on the meaning of knowledge, avoiding the semantic ambiguity that may occur in traditional keyword retrieval; and relational metadata can expand the search scope based on the relationships between knowledge, finding other knowledge paragraphs related to the current knowledge, thereby improving the comprehensiveness and accuracy of the retrieval.

[0179] Step S405: Determine the knowledge segment to which the knowledge metadata belongs.

[0180] In a multimodal knowledge base, each knowledge segment is stored in association with its knowledge metadata. Accordingly, the knowledge segment to which each knowledge metadata belongs can be queried in the multimodal knowledge base.

[0181] Step S406: Using a large language model, based on the knowledge metadata and the knowledge paragraph to which the knowledge metadata belongs, generate the answer to the target question.

[0182] In this embodiment, knowledge metadata provides richer retrieval dimensions. Compared to directly retrieving knowledge paragraphs, it can utilize attributes such as topics and keywords to more accurately find knowledge closely related to the target subtask, avoiding the problem of overly broad or inaccurate search results, and improving the accuracy and efficiency of knowledge paragraph acquisition. Furthermore, since knowledge metadata provides key information about the knowledge, while knowledge paragraphs provide detailed content, generating answers based on knowledge metadata and their respective knowledge paragraphs allows large language models to fully utilize the knowledge in the multimodal knowledge base, generating more targeted, logical, and coherent answers, thus improving the quality and credibility of the responses.

[0183] Based on the first embodiment described above, a fifth embodiment of this application is proposed. Contents that are the same as or similar to the first embodiment can be referred to the above description and will not be repeated hereafter. (Refer to...) Figure 6 In the fifth embodiment, step S40 includes steps S407 to S409:

[0184] Step S407: Generate the current problem-solving steps based on the problem and multiple target sub-tasks using a large language model.

[0185] The current problem-solving steps are specific operations generated by the large language model based on the target problem and multiple target sub-tasks, which are used to progressively advance the solution to the problem. These operations include retrieving specific types of knowledge.

[0186] Step S408: Execute the current problem-solving steps to obtain the problem-solving result.

[0187] Optionally, the current problem-solving step includes the execution tool to be invoked and the invocation parameters of the execution tool. Accordingly, executing the current problem-solving step to obtain the problem-solving result includes: invoking the execution tool based on the invocation parameters and obtaining the invocation result; and generating the problem-solving result of the current problem-solving step based on the invocation result through a large language model.

[0188] An execution tool refers to a variety of tools or functional modules that can be invoked to complete a specific task or processing step. These tools can be of different types, such as data retrieval tools, calculation tools, and analysis tools. Invocation parameters are a set of information specifying how to invoke the execution tool. These parameters tell the execution tool the specific operational requirements, input data, etc. For example, if the execution tool is a search engine, the invocation parameters may include search keywords, search scope, sorting method, etc.; if it is a mathematical calculation tool, the invocation parameters may include specific calculation values ​​and operation types.

[0189] Large language models may suffer from inefficiency or excessive resource consumption when handling certain tasks. Execution tools, on the other hand, are optimized for specific tasks and can complete the work more efficiently. By distributing some tasks to execution tools, large language models can focus on analyzing and integrating the call results to generate the final problem-solving outcome. This leverages the strengths of each tool, optimizes resource utilization, and reduces the overall operating cost of the system.

[0190] Optionally, the execution tools include at least one of the following: domain-specific large models for different knowledge domains, database query tools, business system interface calling tools, and search engines.

[0191] Domain-specific large-scale models are large-scale language models specifically trained for a particular knowledge domain. They possess more specialized capabilities in understanding, processing, and generating knowledge within that domain. Taking an enterprise intelligent assistant system as an example, the domain-specific large-scale model can include: a product expert model, responsible for querying product functions and technical parameters; an after-sales expert model, responsible for answering product-related questions and maintenance / purchase inquiries; a process and system expert model, responsible for answering questions about product sales processes, R&D interaction processes, and quotation approval processes; an enterprise human resources expert model, responsible for human resources-related questions and employee function queries; and a solution expert model, providing cross-product portfolio solution design and generating industry solutions.

[0192] Database query tools are software tools used to retrieve and obtain data from databases. They can extract the required data from the database based on an input query and return it to the model in a specific format.

[0193] A business system interface calling tool is a tool that connects different business systems and enables data interaction and functional collaboration between them by calling their interfaces. Through this tool, one system can request another system to perform specific operations or retrieve relevant data, thereby achieving integration and collaboration between different business systems and improving the automation and efficiency of business processes. Search engines are used to collect information from the internet, organize and process the information, and then return it to models.

[0194] In this embodiment, the domain-specific large model is specially trained to better understand and process domain-specific knowledge, providing more professional and accurate answers to related questions. The database query tool accurately retrieves the required information from large amounts of structured data, providing accurate data support for the answer. The business system interface call tool can obtain real-time data and accurate information from the business system, ensuring that the answer matches the actual business situation. The search engine can find the latest and most comprehensive relevant information from a wide range of online resources, further improving the accuracy and completeness of the answer. By integrating these various types of execution tools, the system can adapt to various complex application scenarios and handle more diverse user questions.

[0195] Step S409: Using the large language model, if it is determined that the question processing result meets the answer conditions, the question processing result is used as the answer to the question; or, if it is determined that the question processing result does not meet the answer conditions, based on the question, multiple subtasks and the obtained question processing results, the next question processing step is regenerated, and the question processing result of the next question processing step is obtained, until the obtained question processing result meets the answer conditions.

[0196] Answer criteria are the standards or guidelines used to determine whether the result of problem processing can serve as the final answer to the target question. These can involve multiple aspects, such as the completeness, accuracy, relevance, and confidence level of the answer. For example, the answer needs to cover all the key points of the question, be closely related to the topic of the question, and the model's confidence level for the answer must reach a certain threshold. Only when the result of problem processing meets these conditions can it be used as the final answer output.

[0197] Optionally, the terminal determines how to handle the target problem based on its complexity. Specifically, the terminal performs intent recognition on the target problem to obtain the intent recognition result. If the intent recognition result determines that the target problem is complex, the terminal searches for a matching reference problem-solving case from the experience knowledge base. Subsequently, a large language model is used to decompose the target problem based on the reference problem-solving case, and an answer is generated by combining the multiple sub-tasks derived from the decomposition. If the intent recognition result determines that the target problem is simple, the terminal selects a target domain large model from the domain model set that matches the intent recognition result, and generates the answer corresponding to the target problem using the target domain large model. The domain model set contains multiple domain large models for different knowledge domains. Each domain large model is specifically trained and optimized for a particular knowledge domain and can handle related problems within that domain.

[0198] Intent recognition refers to using natural language processing technology to analyze the text content of a target question, understand the user's purpose, intent, and needs, and categorize it into different intent categories so that the system can take appropriate actions based on these intents. For example, intent recognition results include information such as the intent to which the question belongs, whether the question falls within the answerable range, whether the question is reasonable, and whether it is clear. Accordingly, if the intent recognition results determine that the target question falls within the answerable range and is reasonable and clear, an answer can be generated based on the target question. Otherwise, the answer is rejected or the user is prompted to re-enter the question.

[0199] For example, if the intent recognition result contains multiple intents or contains a target intent, the target question is determined to be a complex question. Otherwise, the target question is determined to be a simple question. The target intent can be set according to the application scenario. For instance, in a certain application scenario, if a question with a certain intent is difficult to answer, that intent can be preset as the target intent.

[0200] In this embodiment, the target question is categorized into simple and complex questions by performing intent recognition. For simple questions, a target domain big data model matching the intent recognition result is used to generate the answer. This leverages the domain big data model's expertise in a specific domain and its powerful language understanding and generation capabilities to provide accurate and professional answers. For complex questions, a big data language model, using reference question-handling cases from an experience knowledge base, decomposes the question into multiple sub-tasks. Based on these sub-tasks, in-depth analysis and reasoning are performed to generate comprehensive and logically clear answers. This addresses the challenges of multi-domain knowledge fusion and complex logical reasoning in complex questions, thereby improving the accuracy and completeness of answers to complex questions. This approach enables the system to adapt to various types and domains of questions, exhibiting greater flexibility and adaptability.

[0201] Optionally, generating an answer to the target question using a target domain big model includes: using the target domain big model to call other domain big models in the domain model set besides the target domain big model to obtain answer reference information, and generating an answer to the target question based on the answer reference information.

[0202] Reference information refers to various information related to the target question obtained by calling other domain-specific models through the target domain-specific model. This information can include knowledge, perspectives, data, and case studies from different domains, used to assist the target domain-specific model in generating a more comprehensive and accurate answer.

[0203] In traditional model applications, large-scale models in different domains are often trained and applied independently, forming relatively isolated information silos. In this approach, the target domain's large-scale model obtains reference information by calling upon large-scale models in other domains, breaking down these inter-domain barriers. This cross-domain knowledge fusion method allows the target domain's large-scale model to access more diverse types of information and knowledge, enhancing its generalization ability. This enables it to better cope with and handle marginal or cross-domain problems, rather than being limited to its original core training domain, thus broadening the model's application scope and adaptability.

[0204] In this embodiment, by continuously generating problem-solving steps, obtaining corresponding problem-solving results, and evaluating and adjusting based on the results, the next problem-solving step is generated in a targeted manner until a satisfactory answer is obtained. This allows the large language model to progressively and deeply solve the problem, avoiding the one-sidedness or inaccuracy that may occur when generating an answer all at once, thus improving the accuracy and quality of the answer. This approach of generating different processing steps through multiple iterations to gradually explore and solve problems can adapt to various changes and special circumstances. Even if errors or incompleteness are found in previous steps during the processing, steps can be regenerated based on existing results and new analysis, demonstrating strong flexibility and adaptability, and better addressing complex problems in the real world.

[0205] Based on the first embodiment described above, a sixth embodiment of this application is proposed. Contents that are the same as or similar to the first embodiment can be referred to the above description and will not be repeated hereafter. (Refer to...) Figure 7 In the sixth embodiment, after step S40, steps S501 to S502 are further included:

[0206] Step S501: Obtain response feedback information, which includes response quality score and / or content indicating deviations in the response.

[0207] For example, the answer output interface includes a rating control, allowing users to rate the quality of their answers. Another example is a content annotation tool, used to mark any inaccuracies in the answer. Accordingly, the terminal generates answer feedback information based on the user's rating and / or annotation actions.

[0208] Step S502: Based on the response feedback information, perform a knowledge base update operation.

[0209] Knowledge base update operations refer to a series of operations that modify, supplement, or optimize the knowledge base based on feedback information from responses. The aim is to improve the accuracy and usability of the knowledge base, so that it can provide higher-quality and more accurate information when generating subsequent responses. Knowledge base update operations include update operations performed on various types of knowledge bases, such as multimodal knowledge bases and experience-based knowledge bases.

[0210] For example, for content with discrepancies marked in the response feedback or answers with low scores, a knowledge base update reminder is generated. This reminder includes the target question, the corresponding answer, and the response feedback. The reminder is sent to the terminal of the knowledge base maintainer, who can then analyze the knowledge base's shortcomings based on this information and perform updates. Examples include correcting errors or supplementing the knowledge base.

[0211] By updating the knowledge base based on actual response feedback, the knowledge base becomes increasingly comprehensive, and the answers generated by the question-and-answer system become more and more in line with actual needs and user expectations, thereby achieving continuous system optimization and constantly improving response performance.

[0212] Optionally, in addition to updating the knowledge base using user feedback, model self-checking can also be used for knowledge base updates. Specifically, multiple processing samples from the large language model within a preset time period are obtained. These processing samples include the questions processed by the large language model and the corresponding answers. The large language model then filters out processing samples whose answer quality does not meet the quality criteria. Based on the filtered processing samples, a knowledge base update operation is performed. The quality criteria are pre-defined standards or criteria used to measure the quality of the answers generated by the large language model, such as requirements for accuracy, completeness, and logic, used to determine whether the answers in the processing samples meet the quality requirements.

[0213] In this embodiment, the model self-check can proactively examine the processed samples of the large language model within a preset time period, without relying on user feedback to discover answer quality issues. This means that even if users do not provide timely or comprehensive feedback, the system can still promptly identify shortcomings in answer performance, thereby updating the knowledge base in a timely manner and ensuring answer quality.

[0214] Optionally, the multimodal knowledge base can also synchronize data in real time with other business systems via API (Application Programming Interface). Examples include document management systems, wiki systems, and sales systems. This means that when data changes in any business system, these changes are immediately transmitted to the multimodal knowledge base via the API, updating the data in the multimodal knowledge base accordingly, thus achieving dynamic updates to the knowledge base.

[0215] The entire solution proposed in this application includes three main processes: building a multimodal database, handling problems, and updating the database. Figure 8 This is a schematic diagram illustrating a process for constructing a multimodal database as provided in this application. (Reference) Figure 8 The process begins with knowledge acquisition, which involves acquiring new knowledge documents. Next, documents are categorized, meaning metadata is annotated for each document. Then, document content is extracted. If the document contains images, OCR (Optical Character Recognition) is used to identify the text within the images. The images and their text content are then stored. Next, images in the document are replaced with image identifiers, and these images are associated with each other for storage. Documents without images are then segmented, and segmented indexes and vector indexes are built. During the construction of these indexes, metadata is added to each document segment, and a structured index is built using this metadata. Finally, the document is marked as published, completing its storage. It's important to note that the database construction process includes not only the storage of new documents but also the updating of existing documents in the database. That is, the information in existing documents is replaced with the information in the updated document. The processing of updated documents is similar to that of new knowledge documents and will not be elaborated upon here. Additionally, it's worth noting that besides storing and updating knowledge at the document level, it can also store and update knowledge at the document paragraph level. For example, it can correct, supplement, or delete content on document paragraphs in the database, making it easier for knowledge base maintainers to manage the database.

[0216] Figure 9 This is a schematic diagram of a problem-solving process provided in this application. (Reference) Figure 9User input can be text, images, or files. The terminal performs multimodal input processing. If the input is an image, image recognition is performed, and the recognized text is used as the question. If the input is a file, file parsing is performed, and the parsed text is used as the question. Then, intent recognition is performed on the question. Based on the intent recognition result, the complexity of the question is determined. If it is complex, a large language model is used to break it down based on problem-solving cases in an experience knowledge base. Next, the large language model uses the multiple sub-tasks obtained from the problem breakdown to plan the thought chain, that is, to determine the current problem-solving steps. The problem-solving steps include the execution tools to be called and the calling parameters. Then, the execution tools are called using the calling parameters to obtain the call results, and the call results are integrated to obtain the problem-solving result. The execution tools include domain large models, database query tools, business system interface calling tools, search engines, etc. Afterwards, it is determined whether the problem-solving result meets the answer conditions. If not, reflection and reasoning are performed based on the problem-solving result to determine the next problem-solving step. This process is repeated multiple times until the problem-solving result meets the answer conditions.

[0217] Then, the problem-solving results undergo structured output processing, including replacing image identifiers with images and arranging tables, text, and images in the results to obtain a structured answer. It's important to note that when the intent recognition result indicates a simple problem, a domain-wide model is used to process the problem and obtain the solution. This domain-wide model processing involves selecting a target domain-wide model from the domain model set that matches the intent recognition result, and then using this target model to coordinate with other domain-wide models in the set to collaboratively process the problem and obtain the solution. This solution is then similarly processed into a structured output to obtain the answer to the question.

[0218] Figure 10 This is a schematic diagram illustrating a database update process provided in this application. (Reference) Figure 10After outputting an answer, the system checks for feedback information. If feedback is present, it records the feedback and performs manual review. If the feedback indicates insufficient knowledge base information, a knowledge update process is triggered. If no feedback is found, the model self-checks for answers that do not meet quality requirements within a preset time period. If yes, the knowledge update process is triggered; otherwise, it ends. The knowledge base update process includes sending a notification to knowledge base maintenance personnel, prompting them to perform the corresponding update operations. It should be noted that the model does not need to perform self-checks every time an answer is generated, but can perform them periodically within a preset time period. For example, once a day or once a week; this embodiment does not impose such limitations.

[0219] The following concrete example illustrates how the solution in this application addresses complex problems in real-world scenarios. Assume this solution is applied to an enterprise intelligent assistant system. Users, such as a company's sales and solutions architects, can upload user requirement documents to the enterprise intelligent assistant system and ask questions in natural language, such as, "Referring to the user requirements in the document, provide a solution and a corresponding quote."

[0220] Accordingly, the enterprise intelligent assistant system first parses the user-uploaded requirement document, integrating the parsed user requirements with the user's question to obtain the target question. Next, it performs intent recognition on the target question to determine whether it is a simple or complex question. If it is a simple question, such as product information search, contact person lookup, or process inquiry, it directly identifies the target domain model that matches the intent recognition result and generates an answer using the target domain model. If it is a complex question, it enters the problem decomposition stage, that is, it searches for reference problem-solving cases of similar questions in the experience knowledge base, and uses a large language model based on these reference problem-solving cases to decompose the target question into a task plan table, which contains multiple sub-tasks.

[0221] Then, the target problem, task schedule, and descriptions of the various execution tools in the system are input into the large language model. The large language model outputs the current problem-solving steps: Locate product information and contact persons, the execution tools to be called (product information query expert model and contact person query interface call tool), and the call parameters (product name and function name). Based on these call parameters, these two execution tools are called to obtain the call results (product introduction information and relevant contact information). Next, the previous input and the call results are input into the large language model to obtain the current problem-solving steps: Generate a solution, the execution tool to be called (solution expert model), and the call parameters (product information and user requirements). Based on these call parameters, the execution tool is called to obtain the call results (solution content). Next, the previous input and the call results are input into the large language model to obtain the current problem-solving steps: Continue to obtain quotation-related information, the execution tools to be called (enterprise process expert model and product specification database query tool), and the call parameters (function name and product name). Based on the call parameters, these two execution tools are invoked, yielding the call results (specifications, product model, purchase duration and quantity, quotation process, etc.). Next, the previous input and the call results are input into the large language model, resulting in the model's output of the current problem-solving steps: execute the quotation task, the execution tool to be invoked (quotation expert model), and the call parameters (specifications, product model, purchase duration and quantity, etc.). Then, based on these parameters, the execution tool is invoked, obtaining the quotation number. Finally, the previous input and the call results are input into the large language model, which integrates the obtained data and outputs the answer.

[0222] Then, the enterprise intelligent assistant system performs structured output processing on the responses output by the large language model through the output processing module. The output processing module includes:

[0223] The rich text processing module is responsible for converting image identifiers referenced in answers into corresponding images, or converting image identifiers into links to corresponding images. This allows users to easily access relevant resources by clicking on links, improving the interactivity of the answers.

[0224] The file processing module can convert user answers into Word files (a file format) and provide a download link. This allows users to download, save, view, edit, or print the files offline, meeting their needs for document format and local storage.

[0225] Page processing module: This module provides direct access to the results page or buttons for redirecting to results queries. This allows users to directly view content provided by third-party tools within the answers, or to jump to the third-party tool's system via the jump buttons to view related content.

[0226] After the three output processing modules described above perform structured output processing on the large language model's response, a complete solution including a flowchart and product architecture diagram, along with a quotation number, is obtained. This quotation number can be used as a jump button in third-party tool quotation systems; clicking the quotation number will redirect users to the quotation system to view the corresponding quotation.

[0227] In traditional solutions, a company's sales and solutions architects need a comprehensive understanding of product knowledge, pricing processes, and the ability to use pricing systems to provide solutions and pricing information to users. Given the large number of products and the vast amount of product data within a company, coupled with complex pricing processes, acquiring this knowledge to provide services to users is extremely challenging. However, the solution provided in this application eliminates the need for sales and solutions architects to possess such extensive knowledge to provide solutions and pricing services to users. Furthermore, the system can complete the provision of solutions and pricing information that would take hours of human labor, all within a minute. This significantly improves task efficiency and saves companies labor costs.

[0228] Another point to note is that the above examples are only for understanding this application and do not constitute a limitation on the answer generation method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0229] This application also provides an answer generation device, please refer to... Figure 11 The answer generation device includes:

[0230] Question acquisition module 10 is used to acquire the target question corresponding to the dialogue command in response to the dialogue command;

[0231] The case search module 20 is used to search for reference problem-solving cases that match the target problem from the experience knowledge base based on the target problem. The experience knowledge base includes multiple problem-solving cases, and each problem-solving case includes a problem and multiple sub-tasks decomposed from the problem.

[0232] Problem decomposition module 30 is used to decompose the target problem into multiple target sub-tasks based on reference problem handling cases using a large language model;

[0233] The answer generation module 40 is used to generate answers to target questions based on multiple target sub-tasks.

[0234] Optionally, the device further includes:

[0235] The case addition module is used to respond to a case addition command, obtain the problem corresponding to the case addition command and the multiple sub-tasks broken down from the problem; generate a problem handling case based on the problem corresponding to the case addition command and the multiple sub-tasks broken down from the problem; and store the generated problem handling case in the experience knowledge base.

[0236] Optionally, the case addition module is used to respond to a case addition command and retrieve the question edited in the case editing interface, as well as multiple subtasks broken down from the question.

[0237] Optionally, the answer generation module 40 is used to retrieve multiple knowledge paragraphs from a multimodal knowledge base based on multiple target sub-tasks, wherein each knowledge paragraph contains an image identifier; generate an answer containing the image identifier based on the multiple knowledge paragraphs using a large language model; and replace the image identifier in the answer with the image corresponding to the image identifier to obtain the answer corresponding to the target question.

[0238] Optionally, the device further includes:

[0239] The knowledge base construction module is used to respond to knowledge addition instructions, retrieve the document corresponding to the knowledge addition instruction, replace the images in the document with the image identifiers corresponding to the images to obtain the document in plain text form, divide the plain text document into multiple knowledge paragraphs, and store the multiple knowledge paragraphs in the multimodal knowledge base.

[0240] Optionally, the knowledge base construction module is also used to associate images with their corresponding image identifiers and store them in a multimodal knowledge base;

[0241] The answer generation module 40 is also used to query the image corresponding to the image identifier from the multimodal knowledge base.

[0242] Optionally, the answer generation module 40 is used to retrieve knowledge metadata that matches multiple target sub-tasks from a multimodal knowledge base; determine the knowledge paragraph to which the knowledge metadata belongs; and generate an answer corresponding to the target question based on the knowledge metadata and the knowledge paragraph to which the knowledge metadata belongs, using a large language model.

[0243] Optionally, the knowledge metadata includes at least one of the following metadata:

[0244] Basic metadata includes at least one of the following: knowledge type, knowledge source, applicable scenario, and update time of the knowledge segment to which it belongs;

[0245] Semantic metadata contains semantic information about the knowledge segment to which it belongs;

[0246] Related metadata, which includes the relationship between the knowledge paragraph and other knowledge paragraphs.

[0247] Optionally, the answer generation module 40 is used to generate the current problem-solving steps based on the question and multiple target sub-tasks using a large language model; execute the current problem-solving steps to obtain the problem-solving result; and, if the problem-solving result satisfies the answer conditions using the large language model, use the problem-solving result as the answer to the question; or, if the problem-solving result does not satisfy the answer conditions, regenerate the next problem-solving step based on the question, multiple sub-tasks, and the obtained problem-solving result, and obtain the problem-solving result of the next problem-solving step, until the obtained problem-solving result satisfies the answer conditions.

[0248] Optionally, the current problem-solving steps include the execution tool to be invoked and the invocation parameters of the execution tool;

[0249] The answer generation module 40 is used to call the execution tool based on the call parameters and obtain the call result; and to generate the problem processing result of the current problem processing step based on the call result through the large language model.

[0250] Optionally, the execution tool includes at least one of the following:

[0251] Domain-specific large models, database query tools, business system interface calling tools, and search engines are available for different knowledge domains.

[0252] Optionally, the device further includes:

[0253] The intent recognition module is used to recognize the intent of the target question and obtain the intent recognition result.

[0254] The case search module 20 is used to search for reference problem-solving cases that match the target problem from the experience knowledge base when the target problem is determined to be a complex problem based on the intent recognition results.

[0255] Optionally, the answer generation module 40 is also used to select a target domain large model that matches the intent recognition result from the domain model set when the target question is determined to be a simple question based on the intent recognition result. The domain model set contains multiple domain large models for different knowledge domains; and generate the answer corresponding to the target question through the target domain large model.

[0256] Optionally, the answer generation module 40 is used to obtain answer reference information by calling other domain models in the domain model set other than the target domain model through the target domain model, and generate the answer corresponding to the target question based on the answer reference information.

[0257] Optionally, the device further includes:

[0258] The first update module is used to obtain answer feedback information, which includes the answer quality score and / or the content marked as having deviations in the answer; based on the answer feedback information, the knowledge base update operation is performed.

[0259] Optionally, the device further includes:

[0260] The second update module is used to obtain multiple processing samples of the large language model within a preset time period. The processing samples include the questions processed by the large language model and the corresponding answers. The large language model is used to filter out the processing samples whose answer quality does not meet the quality conditions. Based on the filtered processing samples, the knowledge base update operation is performed.

[0261] Optionally, the device further includes:

[0262] The problem rewriting module is used to rewrite the target problem. The rewriting process includes at least one of the following:

[0263] Based on historical dialogue, supplement the content of the target question;

[0264] Remove meaningless words from the target question;

[0265] Replace the words in the target question.

[0266] The answer generation device provided in this application, employing the answer generation method in the above embodiments, can solve the technical problem in related technologies where large language models struggle to accurately grasp the key points of complex problems and perform effective reasoning, resulting in low accuracy of answers. Compared with the prior art, the beneficial effects of the answer generation device provided in this application are the same as those of the answer generation method provided in the above embodiments, and other technical features in the answer generation device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0267] This application provides an answer generation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the answer generation method in Embodiment 1 above.

[0268] The following is for reference. Figure 12The diagram illustrates a structural schematic suitable for implementing the answer generation device of the embodiments of this application. The answer generation device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 12 The answer generation device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0269] like Figure 12 As shown, the response generation device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the response generation device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the response generating device to communicate wirelessly or wiredly with other devices to exchange data. Although response generating devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0270] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0271] The answer generation device provided in this application, employing the answer generation method described in the above embodiments, can solve the technical problem in related technologies where large language models struggle to accurately grasp the key points of complex problems and perform effective reasoning, resulting in low accuracy of answers. Compared with the prior art, the beneficial effects of the answer generation device provided in this application are the same as those of the answer generation method provided in the above embodiments, and other technical features of this answer generation device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0272] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0273] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0274] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the answer generation method in the above embodiments.

[0275] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0276] The aforementioned computer-readable storage medium may be included in the response generation device; or it may exist independently and not assembled into the response generation device.

[0277] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the answer generation device, cause the answer generation device to: in response to a dialogue instruction, acquire a target question corresponding to the dialogue instruction; based on the target question, search for reference problem-solving cases matching the target question from an experience knowledge base, the experience knowledge base including multiple problem-solving cases, each problem-solving case including a question and multiple sub-tasks decomposed from the question; using a large language model, decompose the target question into multiple target sub-tasks based on the reference problem-solving cases; and generate an answer to the target question based on the multiple target sub-tasks.

[0278] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0279] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0280] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0281] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described answer generation method. This addresses the technical problem in related technologies where large language models struggle to accurately grasp the key points of complex problems and perform effective reasoning, leading to low accuracy in answers. Compared to existing technologies, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the answer generation method provided in the above embodiments, and will not be elaborated upon here.

[0282] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the answer generation method described above.

[0283] The computer program product provided in this application can solve the technical problem in related technologies where large language models struggle to accurately grasp the key points of complex problems and perform effective reasoning, resulting in low accuracy of answers. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the answer generation method provided in the above embodiments, and will not be repeated here.

[0284] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for generating answers, characterized in that, The method includes: In response to a dialogue command, obtain the target question corresponding to the dialogue command; Based on the target problem, a reference problem-solving case matching the target problem is searched from the experience knowledge base. The experience knowledge base includes multiple problem-solving cases, and each problem-solving case includes a problem and multiple sub-tasks decomposed from the problem. Using a large language model, based on the reference problem processing case, the target problem is decomposed into multiple target sub-tasks; Based on the multiple target sub-tasks, an answer to the target question is generated.

2. The method as described in claim 1, characterized in that, Before searching for reference problem-solving cases matching the target problem from the experience knowledge base based on the target problem, the method further includes: In response to a case addition command, obtain the question corresponding to the case addition command and multiple sub-tasks decomposed from the question; Based on the case, add the corresponding problem to the instruction and the multiple sub-tasks decomposed from the problem to generate a problem handling case; The generated problem-solving cases are stored in the experience knowledge base.

3. The method as described in claim 2, characterized in that, The step of responding to a case addition instruction by obtaining the question corresponding to the case addition instruction and multiple subtasks decomposed from the question includes: In response to the case addition command, retrieve the question edited in the case editing interface and the multiple subtasks decomposed from the question.

4. The method as described in claim 1, characterized in that, The step of generating an answer to the target question based on the multiple target sub-tasks includes: Based on the multiple target sub-tasks, multiple knowledge segments are retrieved from the multimodal knowledge base, and the knowledge segments contain image identifiers; Using the large language model, and based on the multiple knowledge segments, an answer containing the image identifier is generated; Replace the image identifier in the answer with the image corresponding to the image identifier to obtain the answer to the target question.

5. The method as described in claim 4, characterized in that, Before retrieving multiple knowledge segments from the multimodal knowledge base based on the multiple target sub-tasks, the method further includes: In response to a knowledge addition instruction, retrieve the document corresponding to the knowledge addition instruction; Replace the images in the document with the image identifiers corresponding to those images to obtain a plain text document. The plain text document is divided into multiple knowledge segments, and the multiple knowledge segments are stored in the multimodal knowledge base.

6. The method as described in claim 5, characterized in that, The method further includes: The images are associated with their corresponding image identifiers and stored in the multimodal knowledge base; Before replacing the image identifier in the answer with the image corresponding to the image identifier, the method further includes: Retrieve the image corresponding to the image identifier from the multimodal knowledge base.

7. An answer generation device, characterized in that, The device includes: The question acquisition module is used to acquire the target question corresponding to the dialogue command in response to the dialogue command; The case search module is used to search for reference problem-solving cases that match the target problem from the experience knowledge base based on the target problem. The experience knowledge base includes multiple problem-solving cases, and each problem-solving case includes a problem and multiple sub-tasks decomposed from the problem. The problem decomposition module is used to decompose the target problem into multiple target sub-tasks based on the reference problem processing case using a large language model; The answer generation module is used to generate an answer to the target question based on the multiple target sub-tasks.

8. An answer generation device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the answer generation method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the answer generation method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the answer generation method as described in any one of claims 1 to 6.

Images