Interaction information processing method, electronic equipment and program product
By iteratively obtaining supplementary search results, the problem of incomplete search results when large language models answer questions is solved, resulting in more accurate answers, improved user experience, and enhanced ability to answer complex questions.
Patent Information
- Application Number
- CN202511028859.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-24
- Publication Date
- 2025-10-28
AI Technical Summary
When large language models rely on search results to answer questions, incomplete search results may lead to inaccurate answers, and there is a lack of verification mechanisms for the completeness of search results.
By iteratively obtaining supplementary search results until the search results are complete, the completeness of the search results is predicted using a large language model, and supplementary searches are performed when the results are incomplete, thus generating complete search results to improve the accuracy of the answers.
It improves the accuracy of large language models in answering questions, enhances the user experience in human-computer interaction scenarios, and can handle complex problems with complex logical structures and the need for multiple rounds of information supplementation.
Smart Images
Figure CN120849567A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to artificial intelligence technology, and in particular to an interactive information processing method and electronic devices and software products. Background Technology
[0002] When performing question-answering tasks, Large Language Models (LLMs) sometimes need to call tools such as search engines to retrieve information related to the user's original question from external knowledge bases or websites, and then rely on the retrieved information (i.e., the search results) to answer the user's original question. If there are problems with the search results returned by the search engine, the answer given by the Large Language Model may be inaccurate. Summary of the Invention
[0003] To address the aforementioned technical problems, this disclosure is proposed. Embodiments of this disclosure provide an interactive information processing method, an electronic device, and a program product.
[0004] According to one aspect of the present disclosure, an interactive information processing method is provided, comprising: in response to receiving question information, obtaining a first search result corresponding to the question information; iteratively performing the following operations: in the Nth iteration, determining whether the existing search results lack necessary information required to reply to the question information; N≥1, and N is an integer; in response to the existing search results lacking necessary information required to reply to the question information, obtaining corresponding supplementary search results for the missing necessary information in the existing search results, and entering the next iteration; or, in response to the current existing search results not lacking necessary information required to reply to the question information, ending the iteration; generating reply information to the question information based on the current existing search results, and outputting the reply information; wherein, the current existing search result in the first iteration is the first search result; when N≥2, the existing search results in the Nth iteration include the existing search results in the previous iteration and the supplementary search results obtained in the previous iteration.
[0005] According to another aspect of the embodiments of this disclosure, an interactive information processing apparatus is provided, comprising: a first acquisition module, configured to acquire a first search result corresponding to the question information in response to receiving question information; an iteration module, configured to iteratively perform the following operations: in the Nth iteration, determining whether the existing search results lack necessary information required to reply to the question information; N≥1, and N is an integer; in response to the current existing search results lacking necessary information required to reply to the question information, acquiring corresponding supplementary search results for the currently missing necessary information, and entering the next iteration; or, in response to the current existing search results not lacking necessary information required to reply to the question information, ending the iteration; wherein, the existing search results in the first iteration are the first search results; when N≥2, the existing search results in the Nth iteration include the existing search results in the previous iteration and the supplementary search results acquired in the previous iteration; and an output module, configured to generate reply information to the question information based on the current existing search results, and output the reply information.
[0006] Based on the interactive information processing method and apparatus provided in the above embodiments of this disclosure, after obtaining the first search result corresponding to the original question, the first search result is used as the initial value of the current existing search results. Iterative operations are performed to predict the completeness of the current search results and to supplement them if the prediction is incomplete, until the current existing search results are complete and do not lack the necessary information to answer the user's question. That is, after at least one round of completeness prediction and search result supplementation, a complete search result is obtained. In scenarios where answers rely on search results, complete search results can improve the accuracy of large language models in answering questions. This method overcomes the deficiency in related technologies where large language models lack self-checking of the completeness of search results when answering questions. By detecting and supplementing the completeness of search results, the probability of inaccurate answers due to incomplete search results can be reduced, improving the user experience in human-computer interaction scenarios.
[0007] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0008] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0009] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein: Figure 1 This is a schematic diagram of the system architecture of one embodiment of the interactive information processing method disclosed herein; Figure 2 This is a flowchart of one embodiment of the interactive information processing method disclosed herein; Figure 3 A flowchart of an embodiment of the interactive information processing method of this disclosure performing two rounds of iteration; Figure 4 A flowchart of one embodiment of the interactive information processing method of this disclosure, which performs three rounds of iteration; Figure 5 This is a flowchart of an optional embodiment of the interactive information processing method disclosed herein for determining whether existing search results are missing information; Figure 6 This is a flowchart of another optional embodiment of the interactive information processing method disclosed herein for determining whether the current existing search results are missing information; Figure 7 This is a flowchart of yet another optional embodiment of the interactive information processing method disclosed herein for determining whether the current existing search results are missing information; Figure 8 A flowchart of an optional embodiment of the method disclosed herein, which outputs response information after obtaining complete search results; Figure 9 This is a schematic diagram of an optional embodiment of the interactive information processing device disclosed herein; Figure 10 This is a schematic diagram of another optional embodiment of the interactive information processing apparatus of this disclosure; Figure 11 This is a schematic diagram of the structure of the electronic device disclosed herein. Detailed Implementation
[0010] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.
[0011] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of this disclosure.
[0012] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.
[0013] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.
[0014] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.
[0015] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship. The data referred to in this disclosure can include unstructured data such as text, images, and videos, as well as structured data.
[0016] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.
[0017] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.
[0018] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.
[0019] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification.
[0020] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0021] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.
[0022] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.
[0023] Application Overview In the process of realizing this disclosure, the inventors discovered that in current question-answering scenarios based on large language models, a combination of large language models and web search is sometimes used to complete the question-answering process. When answering questions, the large language model relies on the retrieval results. For example, in question-answering scenarios based on Retrieval-Augmented Generation (RAG) technology, text fragments (retrieval results) related to the user's question are first retrieved from an external knowledge base. Then, the retrieved text fragments are used together with the original question input by the user to assist the large language model in generating the answer.
[0024] This approach, which relies on search results, has at least the following problems: If there are problems with the search results, such as incomplete results (meaning the search results are missing some necessary information compared to all the information needed to answer the user's original question), the answer given by the large language model may be inaccurate.
[0025] For example, if a user asks, "Who is the master of X-Zha's master's master's master?" the search engine might only give the result "X-Zha's master is XX Zhenren", missing information about who XX Zhenren's master is, or missing information about XX Zhenren's master's master's master.
[0026] Incomplete search results may stem from the problem of information silos, such as search engines returning results only from a single source, failing to guarantee information completeness. Furthermore, related technologies lack verification mechanisms and automated verification processes for the completeness of search results. For example, in the above example, the search result obtained is "Xzha's master is XX Zhenren," and current technologies lack a mechanism to verify the completeness of this search result.
[0027] In related technologies, some large language model question-answering scenarios are built using a static processing architecture. Static processing architectures generally cannot handle complex problems that require multiple rounds of information supplementation.
[0028] Alternatively, the search results may have other problems, such as inaccurate results, which could also cause large language models to have difficulty providing correct answers.
[0029] Exemplary System The interactive information processing method provided in this exemplary embodiment can be applied to... Figure 1 The exemplary system architecture shown includes a terminal device 10, a search engine server 11, and a cloud server 12 deployed with a large language model.
[0030] based on Figure 1 In the question-and-answer scenario of the system architecture shown, the user inputs their question (hereinafter referred to as the original question) through a terminal device. Upon receiving the user's input of the original question (i.e., question information), the terminal device 10 sends a search request to the search engine server 11, which provides search services. In response to this search request, the search engine server 11 returns search results to the terminal device 10. Based on the user's input of the original question and the returned search results, the terminal device 10 generates a prompt. The prompt includes the search results, the original question, and instructions on what task the large language model should perform. In this embodiment, the task that the large language model needs to perform is at least: predicting whether the returned search results are missing information relative to all the information needed to answer the original question, that is, at least predicting the completeness of the currently obtained search results. The terminal device 10 transmits the generated prompt to the cloud server 12 through the calling interface or interaction interface provided by the large language model. The large language model is deployed on the cloud server 12. In response to the prompt message uploaded by terminal device 10, cloud server 12 executes the task in the prompt message and outputs output data in a specified format. The output data includes the prediction result of the large language model on whether the current search results are complete. That is, the large language model determines whether the current search results lack the necessary information to answer the original question raised by the user. If the prediction result indicates that it is incomplete, other search results need to be obtained again to supplement the incomplete search results.
[0031] Figure 1The system architecture and corresponding explanations shown are merely examples. This scenario illustrates how to determine whether the current search results lack the necessary information to answer the user's original question by obtaining prediction results through a large language model. In reality, other technical means can also be used to determine whether the current search results lack the necessary information to answer the user's original question. For example, predictions can be made using other neural network models with natural language processing or prediction capabilities, or by algorithms capable of performing the aforementioned prediction functions. The following explanation uses obtaining prediction results through a large language model as an example.
[0032] To avoid confusion, the search result corresponding to the original question is referred to as the first search result. In some embodiments, the large language model can also predict the necessary information missing from the first search result relative to all the information required to answer the user's original question. The prediction result may also include the predicted missing necessary information, which can also be referred to as missing information. To avoid confusion, the necessary information missing from the first search result relative to the complete search result is defined as the first necessary information. The cloud server 12 can return the information on whether the predicted first search result is complete and the predicted specific missing first necessary information to the terminal device 10. The terminal device 10 then sends a search request to the search engine server 11 again, requesting to retrieve the text fragments and other content corresponding to the first necessary information. The search result corresponding to the first necessary information returned by the search engine server 11 is defined as the first supplementary search result. After receiving the first supplementary search result, the terminal device 10 combines the first search result and the first supplementary search result. Next, it verifies whether the combined search result (i.e., the current existing search result) is complete. This process is iterated until the large language model predicts that the current search result is complete. The complete search results obtained can be input into the large language model along with the original question to obtain a more accurate answer, and the answer will be presented to the user on the interactive interface of the terminal device 10.
[0033] Figure 1 The terminal device 10 in the example is only. The terminal device 10 can be a computer, smartphone, tablet, smart TV / set-top box, wearable device (such as smartwatch, bracelet, etc.), smart home device (smart speaker, camera, robot, etc.), smart vehicle terminal or other electronic device that supports providing artificial intelligence (AI) service access.
[0034] The search engine server 11 or cloud server 12 can be a standalone server or a server cluster.
[0035] Exemplary methods Figure 2 This is a schematic flowchart of an exemplary embodiment of the present disclosure providing an interactive information processing method. This embodiment can be applied to electronic devices, such as terminal devices. Figure 2 As shown, this method may include the following steps: Step 201: In response to receiving the problem information, obtain the first search result corresponding to the problem information.
[0036] For example, if a user enters the question information (original question) "Who is the master of X-Zha's master's master?", the first search result obtained is "X-Zha's master is XX Zhenren".
[0037] For example, terminal device 10 can call the application programming interface (API) of the search engine to provide retrieval services, and carry the original question input by the user in the call request to realize the call to the search engine, or through the interactive interface provided by the search engine, input the original question as the query content, retrieve information such as text fragments related to the original question, and obtain the first search result.
[0038] Next, steps 202 and 203 are executed iteratively.
[0039] Step 202: In the Nth iteration, determine whether the existing search results lack the necessary information required to answer the question. If yes, proceed to step 203; otherwise, proceed to step 204.
[0040] Where N is an integer, and N≥1. In the first iteration, the current existing search results are the first search results. It is determined whether the current existing search results lack the necessary information to answer the question, that is, whether the first search results lack the necessary information to answer the question.
[0041] Step 202 is a step of detecting or predicting the completeness of the current search results, which involves determining whether the existing search results lack the necessary information to answer the question.
[0042] For example, the first search result is "X-Zha's master is XX Zhenren", which lacks necessary information compared to the original question "Who is the master of X-Zha's master's master?" In other words, the first search result is incomplete and needs to be supplemented.
[0043] Step 203: In response to the lack of necessary information for answering the question in the existing search results, obtain the corresponding supplementary search results for the currently missing necessary information, and proceed to the next iteration.
[0044] Supplementary search results are used to supplement existing search results. For example, in the first iteration, after searching based on the first necessary information missing from the first search result, a first supplementary search result is obtained. The first supplementary search result is combined with the first search result to obtain the first existing search result. Then, in the second iteration, the process returns to step 202 to continue predicting whether the currently obtained first existing search result is complete.
[0045] When N≥2, the current search results in this iteration include the search results from the previous iteration and the supplementary search results obtained in the previous iteration. For example, in the second iteration, the current search results include the first search result and the first supplementary search result from the first iteration.
[0046] In step 203, before obtaining the corresponding supplementary search results for the currently missing necessary information, it is necessary to first obtain the currently missing necessary information. For example, a large language model can be used to predict which specific necessary information is missing from the current existing search results relative to all the information required to answer the user's original question. Alternatively, other neural network models or algorithm tools with natural language processing capabilities that support the above prediction function can be used to obtain the necessary information missing in this iteration (referred to as missing information).
[0047] For example, continuing with the above example, the original question is "Who is the master of X-Zha's master's master?" The first search result is "X-Zha's master is XX Zhenren". The first search result is missing the necessary information "Who is the master of XX Zhenren's master's master?" This necessary information can be obtained by calling a large language model, etc.
[0048] After obtaining the missing information corresponding to the current search results, you can continue to call the search engine to search for the missing information and obtain the corresponding first supplementary search results.
[0049] For example, for the missing information "Who is the master of XX Zhenren's master?", the first supplementary search result obtained by searching is "XX Zhenren's master is XX Tianzun".
[0050] In this iteration, the supplementary search results need to be combined with the existing search results from the previous round. For the first iteration, the supplementary search results are combined with the first search result. For example, the first search result "Xzha's master is XX Zhenren" is combined with the first supplementary search result "XX Zhenren's master is XX Tianzun". The combined first existing search result is "Xzha's master is XX Zhenren, XX Zhenren's master is XX Tianzun".
[0051] It should be noted that, in order to prevent confusion and distinguish between different iterations, the various technical concepts involved in each iteration are prefixed with "first", "second", etc. for differentiation. For example, the search result corresponding to the original question is defined as the first search result, the missing necessary information corresponding to the first search result is the first necessary information (or the first missing information), the search result obtained based on the first necessary information is the first supplementary search result, and so on. "First", "second", etc. can be understood as the identifier of the same technical feature in different iterations.
[0052] Step 204: In response to the fact that the current search results do not lack the necessary information to answer the question, the iteration ends; based on the current search results, a response to the question is generated and output.
[0053] The existing search results do not lack the necessary information to answer the question, indicating that the existing search results are complete and can be used to answer the original question raised by the user.
[0054] For example, after three rounds of iteration, two rounds of supplementation were implemented for the first search result, and the complete search result was obtained as "X Zha's master is XX Zhenren, XX Zhenren's master is XX Tianzun, XX Tianzun's master is XX Laozu". Based on this complete search result, the user's original question was answered and the answer information was "X Zha's master's master's master is XX Laozu".
[0055] Figure 2 In the illustrated embodiment, after obtaining the first search result corresponding to the original question, in one iteration, the completeness of the current search result is predicted. If the prediction indicates that the current search result is incomplete, the missing necessary information is retrieved to obtain corresponding supplementary search results, thus supplementing the incomplete search result. After at least one round of iteration, at least one round of supplementation of the incomplete search result can be achieved, thereby gradually obtaining a complete search result. Generating answer information (i.e., reply information) based on the complete search results can improve the accuracy of answering questions in human-computer interaction and enhance the user experience. In addition, multiple rounds of iterative supplementation of search results can also handle complex questions with complex logical structures that require multiple rounds of information supplementation, improving the ability to answer complex questions in human-computer interaction scenarios.
[0056] In practical applications, for relatively simple original questions, a complete search result can be obtained through one round of supplementation. In an optional embodiment, based on Figure 2 The multi-round iterative supplementation mechanism shown is as follows: Figure 3 As shown, this optional embodiment may specifically include the following process: Step 301: In the first iteration, determine that the first search result lacks the necessary information required to answer the question.
[0057] For example, if a user enters the question "How many city blocks are there in the capital of country A?", the first search result will be "The capital of country A is city C", which lacks the necessary information "How many city blocks are there in city C".
[0058] Step 302: In response to the lack of necessary information in the first search result, obtain the first supplementary search result for the first necessary information missing in the first search result, and proceed to the second iteration.
[0059] A search for "How many city blocks are there in City C?" yielded the first supplementary search result "City C has M city blocks".
[0060] Step 303: In the second iteration, determine that the first existing search results do not lack the necessary information required to answer the question.
[0061] The first existing search results include the first search result and the first supplementary search result. For example, the first existing search results can be obtained by concatenating or combining the first search result and the first supplementary search result.
[0062] For example, the first search result "The capital of country A is city C" combined with the first supplementary search result "City C has M blocks" results in the first existing search result "The capital of country A is city C, and city C has M blocks".
[0063] Step 304: In response to the fact that the first existing search result does not lack the necessary information, the iteration ends, and a response message for the question information is generated based on the first existing search result and output.
[0064] For example, if the first existing search result is "The capital of country A is city C, and city C has M city blocks", which does not lack the necessary information, then based on the first existing search result, the original question "How many city blocks are in the capital of country A?" will be answered, and the answer information obtained will be "The capital of country A has M city blocks".
[0065] Figure 3 The illustrated embodiment supplements incomplete search results to obtain complete search results. Based on the complete search results, the user's question can be answered, which can improve the accuracy of question answering in human-computer question answering scenarios.
[0066] In practical applications, for some original questions, complete search results can be obtained through two rounds of supplementation. For example, in one optional embodiment, based on... Figure 2 The multi-round iterative supplementation mechanism shown is as follows: Figure 4 As shown, this optional embodiment may specifically include the following process: Step 401: In the first iteration, determine that the first search result lacks the necessary information required to answer the question.
[0067] For example, if a user enters the question "What is the resident population of the largest district in the capital of country A?", the first search result will be "The capital of country A is city C", which lacks the necessary information "What is the resident population of the largest district in city C?".
[0068] Step 402: In response to the lack of necessary information in the first search result, obtain the first supplementary search result for the first necessary information missing in the first search result, and proceed to the second iteration.
[0069] A search for "How many permanent residents does the largest district in City C have?" yielded the first supplementary search result as "The largest district in City C is District D".
[0070] Step 403: In the second iteration, determine that the first existing search results lack the necessary information required to answer the question.
[0071] For example, the first search result "The capital of country A is city C" combined with the first supplementary search result "The largest district in city C is district D" results in the first existing search result "The capital of country A is city C, and the largest district in city C is district D". This still lacks the necessary information to answer the user's question.
[0072] Step 404: In response to the lack of necessary information in the first existing search results, obtain the corresponding second supplementary search results for the second necessary information that is missing in the first existing search results, and proceed to the third iteration.
[0073] For example, when searching for the missing essential information "How many permanent residents are there in District D?", the second supplementary search result is "District D has 2.1 million permanent residents".
[0074] Step 405: In the third iteration, determine that the second existing search result does not lack the necessary information required to answer the question.
[0075] The second existing search result includes the first existing search result and the second supplementary search result. For example, combining the first existing search result with the second supplementary search result yields the second existing search result: "The capital of country A is city C, the largest district in city C is district D, and district D has a resident population of 2.1 million."
[0076] Step 406: In response to the fact that the current second existing search result does not lack the necessary information required to reply to the question information, the iteration ends, and the reply information is generated based on the second existing search result and output.
[0077] Based on the second existing search result, "The capital of country A is city C, the largest district of city C is district D, and district D has a resident population of 2.1 million", the answer to the user's original question is "The largest district of the capital of country A has a resident population of 2.1 million".
[0078] Figure 4 The illustrated embodiment obtains complete search results by supplementing the search results in two rounds. Based on the complete search results, the user's question is then answered, which can improve the accuracy of answering relatively complex questions in human-computer interaction scenarios.
[0079] In an optional embodiment, in each iteration, it is determined whether the current search results lack the necessary information to answer the question. This can be achieved by calling a large language model to predict or detect the completeness of the search results. Figure 5 As shown, based on Figure 2 In the illustrated embodiment, step 202 may include the following sub-steps: Step 2021: Generate a prompt message.
[0080] In each iteration, the generated prompt includes at least the original question, the search results, and the task instruction (hereinafter referred to as the instruction). The instruction is used to prompt or guide the pre-defined task that the large language model needs to perform. For example, the task that the large language model needs to perform is at least to predict whether the current search results are complete, that is, to predict whether the current search results lack the necessary information to answer the question.
[0081] For example, after obtaining the first search result, a corresponding first prompt is generated based on the original question and the first search result.
[0082] For example, a sample prompt could be as follows: "Your role is a search verifier."
[0083] You will receive a question and several text blocks, which may or may not contain the answer to the question. Your task is to carefully examine these text blocks and provide a response that contains at least one field: 1. status: Whether the retrieved text block contains the answer to the question.
[0084] If the retrieved text block contains the answer to the question, it is 'COMPLETE'; otherwise, it is 'INCOMPLETE'. No other content is required.
[0085] Do not include any other text.
[0086] Context: {retrieved_context} Question: {question} response: " Here, "context" is a variable called the context variable. In the first prompt corresponding to the first round of tasks, the value of the context variable is the first search result. In other words, after obtaining the first search result, the first search result is assigned to the context variable. For example, if the first search result is "X-Zha's master is XX Zhenren", then retrieved_context = X-Zha's master is XX Zhenren. Context can also be called background information.
[0087] The "question" is also a variable, called the question variable. In a multi-round task, the value of the question variable is always equal to the original question input by the user. In other words, after receiving the original question input by the user, the value of the original question can be assigned to the question variable. For example, if the original question is "Who is the master of X-Zha's master's master's master?", then question = Who is the master of X-Zha's master's master's master's master? Step 2022: Input the prompt into the large language model, and the large language model will output at least the state information.
[0088] In each iteration, the prompt is input into the large language model. The output data from the large language model includes at least state information, which indicates whether the current search result lacks necessary information relative to answering the user's original question. Specifically, the state information can be a field whose value indicates whether the current search result lacks necessary information relative to answering the user's original question. For example, in the first iteration, the first output data returned by the large language model (also called the first prediction result) includes the following first state information: "status: 'INCOMPLETE'".
[0089] Here, "status" is an example of the field name for the status information field, and "INCOMPLETE" is the value of the status information field, indicating incompleteness, meaning that the first search result lacks the necessary information to answer the user's original question.
[0090] The data format of the output data returned by the large language model when performing a multi-round prediction task can be the same. For example, the output data returned in each round should at least include a status information field. The value of the status information field is used to indicate whether the current search result obtained in this round of task is missing information relative to the complete search result that is required to answer the original question raised by the user.
[0091] The values of the status information field, or other fields mentioned below, can be of various data types, specifically at least one of the following: integer, real number, character or string, Boolean, etc. For example, a status information field value of "complete" or other characters or numbers representing completeness indicates that no information is missing. A status information field value of "incomplete" or other characters or numbers representing incompleteness indicates that information is missing. For example, the number "0" represents incompleteness, and the number "1" represents completeness. Or, the letter "C" represents completeness, and the letter "I" represents incompleteness, and so on.
[0092] In the first received output data, if the value of the status information field represents "complete," it means that the large language model predicts that the first search result contains all the information needed to answer the user's original question, and the first search result is a complete search result. If the value of the status information field represents "incomplete," it means that the large language model predicts that the first search result is incomplete because it contains some information needed to answer the user's original question, and it needs to be supplemented.
[0093] For example, given the original question "Who is the master of X-Zha's master's master's master?", the first search result is "X-Zha's master is XX Zhenren". The state information field in the prediction result (output data) returned by the large language model is incomplete, so the first search result needs to be supplemented. This can be done by obtaining the first supplementary search result based on the missing essential information. For instance, if the first search result is "X-Zha's master is XX Zhenren", then the missing essential information is "Who is the master of XX Zhenren's master's master?". This missing essential information can be obtained by calling a search engine or searching in a knowledge base with access permissions. For example, a search engine specifically designed for AI applications or an internal knowledge base can be used to search for "Who is the master of XX Zhenren's master's master?" to obtain the first supplementary search result.
[0094] After obtaining the first supplementary search result, the first supplementary search result can be combined with the first search result to obtain the first existing search result.
[0095] Figure 5 The embodiment shown demonstrates that the completeness of search results can be predicted by calling a large language model through the design of prompts, without the need to configure dedicated prediction tools. This facilitates the rapid acquisition of search result completeness analysis results, thereby obtaining complete search results and improving the accuracy of answering questions.
[0096] In an optional embodiment, the task instruction in the prompt is also used to prompt the large language model to output missing information along with the output state information. Missing information refers to the necessary information predicted by the large language model that is missing from the current existing search results relative to the necessary information required to answer the question; for example, the missing information corresponding to the first search result is the first necessary information. Figure 6 As shown, in this optional embodiment, based on Figure 2 In the illustrated embodiment, step 202 may include: Step 2023: Generate a prompt message.
[0097] For example, here is an example of the generated prompt: "Your role is a search verifier."
[0098] You will receive a question and several text blocks, which may or may not contain the answer to the question. Your task is to carefully examine these text blocks and provide a response containing at least two fields: 1. status: Whether the retrieved text block contains the answer to the question.
[0099] If the retrieved text block contains the answer to the question, it is 'COMPLETE'; otherwise, it is 'INCOMPLETE'. No other content is required.
[0100] 2. Missing information: The missing information needed to fully answer the question; it should be concise and direct.
[0101] If no information is missing, set it to an empty string.
[0102] Do not include any other text.
[0103] Context: {retrieved_context} Question: {question} response: " Here, "missing_information" is the field name of the missing information field, and the value of the missing information field is the missing information predicted by the large language model or a specified string. The specified string can be an empty string or other data that indicates that the returned result is empty, such as returning the text "empty" or "none", indicating that the missing information could not be predicted or that the missing information does not exist (i.e., complete).
[0104] Step 2024: Input the prompt into the large language model, and the large language model outputs the status information and missing information.
[0105] For example, if the original question is "Who is the master of X-Zha's master's master?", the first search result is "X-Zha's master is XX Zhenren". The output data returned by the large language model is as follows: 1. Status: 'INCOMPLETE' 2. missing_information: 'Who is the master of XX's master?' "Who is the master of XX Zhenren's master?" is the missing information predicted by the large language model.
[0106] Figure 6 The optional embodiment shown uses the instruction design in the prompts to enable the large language model to not only predict the completeness of the search results, but also to predict the specific missing necessary information. In this way, based on the status information and missing information returned by the large language model, when the status information is incomplete, the missing information can be searched to obtain supplementary search results to supplement the current search results until a complete search result is obtained, thereby improving the accuracy of the large language model in answering questions.
[0107] In one optional embodiment, the task instruction is further used to prompt the large language model to output valid information. Valid information is necessary information extracted from the current existing search results. The large language model can output status information, missing information, and valid information. Furthermore, in the same or other optional embodiments, the prompt may also include specified data format information. For example... Figure 7 As shown, based on Figure 2 In the illustrated embodiment, step 202 may include the following sub-steps: Step 2025: Generate a prompt message.
[0108] In this optional embodiment, the prompt includes question information, current existing search results, instructions, and specified data format information. The specified data format information is used to indicate the format of the data output by the large language model.
[0109] Step 2026: Input the prompt into the large language model, and the large language model outputs status information, valid information and missing information with specified data format.
[0110] For example, in this optional embodiment, an example of the prompt message is as follows: "Your role is a search verifier."
[0111] You will receive a question and several text blocks, which may or may not contain the answer to the question. Your task is to carefully examine these text blocks and provide a JSON response containing at least two fields: 1. status: Whether the retrieved text block contains the answer to the question.
[0112] If the retrieved text block contains the answer to the question, it is 'COMPLETE'; otherwise, it is 'INCOMPLETE'. No other content is required.
[0113] 2. useful_information: Useful information extracted from the retrieved text blocks, which should be concise and direct.
[0114] If no information is missing, set it to an empty string.
[0115] 3. missing_information: Information needed to fully answer the question but missing; should be concise and direct.
[0116] If no information is missing, set it to an empty string.
[0117] Please provide your response in the following format: json { "status":" <status>", "useful_information":"<useful_information> ", "missing_information":"<missing_information> " } Here is an example of the response format: json { "status":"COMPLETE", "useful_information":"The capital of country A is XXX." "missing_information":"The capital of country B" } Do not include any other text.
[0118] Context: {retrieved_context} Question: {question} response: " It includes specified data format information to restrict the format of the output data. For example, "Provide a JSON response with at least two fields" and "Please provide the response in the following format..." give specific requirements for the format of the response (output data) and provide a few examples of response formats.
[0119] The above prompts are merely examples; more specific examples can be derived based on the data structure presented in this disclosure. The language of the prompts can also be English or other languages supported by the large language model.
[0120] It should be noted that in the above embodiments, the combination of existing search results and supplementary search results can be a combination of valid information extracted from existing search results and supplementary search results. For example, valid information extracted from the first search result can be combined with the first supplementary search result.
[0121] In this optional embodiment, by prompting the specific format of the output data, more expected output data can be obtained, making the data structure of the output data clearer and facilitating the quick reading of the required information. For example, by reading the value of a specified field name, key information such as whether it is complete or missing can be quickly obtained.
[0122] Furthermore, the output data returned by the large language model contains useful information extracted from existing search results. Redundant information in the search results can be filtered out by the large language model, reducing interference from invalid other information, which helps to improve the conciseness of the complete search results and reduce the impact of irrelevant information on the accuracy of answering questions.
[0123] In one alternative embodiment, based on Figure 2 Based on the illustrated embodiments, as Figure 8 As shown, step 204 generates and outputs a response to the question based on the existing search results. This process may include the following steps: Step 2041: Generate the complete prompt message.
[0124] A complete prompt includes the complete search results and the original question. The complete search results refer to those retrieved in the last iteration before the end of the iteration, ensuring no necessary information is missing; in other words, it is the current search result in the last iteration. It should be noted that the complete prompt no longer contains instructions to prompt the large language model to perform prediction tasks such as those mentioned in any of the embodiments above, but only instructions to prompt the large language model to answer the question.
[0125] For example, here is a complete example of a prompt: You are a question and answer assistant.
[0126] You will receive a question and several text blocks containing the answer to the question.
[0127] Your task is to carefully read these text blocks and answer the question directly and concisely based on the information provided.
[0128] Please do not include any additional information or comments.
[0129] Question: {question} Context: {retrieved_context} answer:" In this example, the question "Who is the master of X-Zha's master's master?" is followed by the retrieved context "X-Zha's master is XX Zhenren, XX Zhenren's master is XX Tianzun, and XX Tianzun's master is XX Laozu".
[0130] Step 2042: Input the complete prompt into the large language model and receive the response information returned by the large language model.
[0131] For example, the response returned by the large language model is "Xzha's master's master's master is XX Laozu".
[0132] This method utilizes a large language model or other tools to predict the completeness of the initial search results. Through the design of prompts, the large language model is guided to perform a prediction task, at least predicting whether the initial search results are sufficient to answer the original question. If the initial search results are predicted to be insufficient, a first supplementary search result is obtained based on the missing necessary information. This supplementary result is then used to generate a second prompt, which is input into the large language model. Based on the second prediction result returned by the large language model, it is determined whether the first supplementary search result still lacks necessary information. If it is complete, the first supplementary search result is used as the complete search result and input along with the original question into the large language model, outputting the answer to the user's question. A complete search result helps to obtain a more accurate answer, improving the accuracy of answering questions in human-computer interaction scenarios. This method achieves the detection of the completeness of search results, reducing the probability of inaccurate answers due to incomplete search results.
[0133] Any of the interactive information processing methods provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the interactive information processing methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the interactive information processing methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.
[0134] Exemplary device Figure 9 This is a schematic diagram illustrating the structure of one embodiment of the interactive information processing apparatus of this disclosure. The apparatus of this embodiment can be used to implement the corresponding method embodiments of this disclosure. Figure 9 The device shown includes: an acquisition module 1000, an iteration module 1001, and an output module 1002.
[0135] The acquisition module 1000 is used to obtain the first search result corresponding to the received problem information in response to receiving the problem information.
[0136] The iteration module 1001 is used to iteratively perform the following operations: In the Nth iteration, determine whether the current existing search results lack the necessary information required to reply to the question; N≥1, and N is an integer; in response to the current existing search results lacking necessary information, obtain the corresponding supplementary search results for the currently missing necessary information, and enter the next iteration; or, in response to the current existing search results not lacking the necessary information required to reply to the question, end the iteration.
[0137] The output module 1002 is used to generate and output response information to the question information based on the current existing search results.
[0138] Figure 10 This is a schematic diagram of another embodiment of the interactive information processing apparatus disclosed herein.
[0139] The apparatus of this embodiment can be used to implement the corresponding method embodiments of this disclosure, such as... Figure 10 In the device shown, the iteration module 1001 may include a first generation module 10011 and a first calling module 10012.
[0140] When determining whether the current existing search results lack the necessary information to reply to the question, the first generation module 10011 generates a prompt; the prompt includes the question information, the current existing search results, and a task instruction; the task instruction prompts the large language model to output at least status information based on the question information and the current existing search results. The first calling module 10012 inputs the prompt to the large language model, which then outputs at least status information; the status information indicates whether the current existing search results lack the necessary information to reply to the question.
[0141] In an optional embodiment, the iteration module 1001 can specifically perform the following operations: In the first iteration, it is determined that the first search result lacks the necessary information required to reply to the question; in response to the lack of necessary information in the first search result, a first supplementary search result is obtained for the first necessary information missing in the first search result, and a second iteration is performed; In the second iteration, it is determined that the first existing search result does not lack the necessary information required to reply to the question, and the first existing search result includes the first search result and the first supplementary search result; in response to the lack of necessary information in the first existing search result, the iteration ends. The output module 1002 can generate reply information to the question based on the first existing search result and output the reply information.
[0142] In another optional embodiment, the iteration module 1001 can specifically perform the following operations: In the first iteration, it is determined that the first search result lacks the necessary information required to respond to the question information; in response to the lack of necessary information in the first search result, a first supplementary search result is obtained for the first necessary information missing in the first search result, and a second iteration is performed; In the second iteration, it is determined that the first existing search result lacks the necessary information required to respond to the question information, the first existing search result including the first search result and the first supplementary search result; in response to the lack of necessary information in the first existing search result, a corresponding second supplementary search result is obtained for the second necessary information missing in the first existing search result, and a third iteration is performed; In the third iteration, it is determined that the second existing search result does not lack the necessary information required to respond to the question information; the second existing search result includes the first existing search result and the second supplementary search result; in response to the lack of necessary information required to respond to the question information in the second existing search result, the iteration ends. The output module 1002 can generate response information to the question information based on the second existing search result and output the response information.
[0143] In an optional embodiment, the task instruction is further used to prompt the large language model to output missing information; the missing information is the necessary information that the large language model predicts is missing from the current existing search results relative to the necessary information required to reply to the question. The first calling module 10012 can input the prompt into the large language model, and the large language model outputs status information and missing information.
[0144] In an optional embodiment, the task instruction is further used to prompt the large language model to output valid information, which is necessary information extracted from the current existing search results. The first calling module 10012 can receive the status information, missing information, and valid information output by the large language model.
[0145] In an optional embodiment, the prompt also includes specified data format information; the first calling module 10012 can also receive status information, missing information and valid information with specified data format output by the large language model.
[0146] In an optional embodiment, the output module 1002 may include a second generation module and a second invocation module. The second generation module is used to generate a complete prompt; the complete prompt includes the current existing search results and question information. The second invocation module is used to input the complete prompt into the large language model and receive the answer information returned by the large language model.
[0147] Exemplary electronic devices Below, for reference Figure 11 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.
[0148] Figure 11 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.
[0149] like Figure 11 As shown, the electronic device includes one or more processors and memory.
[0150] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.
[0151] The memory can store one or more computer program products, and the memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor can run the computer program products to implement the interactive information processing methods of the various embodiments of this disclosure described above and / or other desired functions.
[0152] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).
[0153] In addition, the input device may also include, for example, a keyboard, a mouse, etc.
[0154] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.
[0155] Of course, to simplify, Figure 11 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.
[0156] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products, including computer program instructions that, when executed by a processor, cause the processor to perform the steps of the interactive information processing methods according to various embodiments of this disclosure as described in the foregoing portions of this specification.
[0157] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of this disclosure. The programming languages include object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0158] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the interactive information processing methods according to various embodiments of this disclosure as described in the foregoing portion of this specification.
[0159] The computer-readable storage medium may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.
[0160] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.
[0161] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.
[0162] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.
[0163] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.
[0164] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.
[0165] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.
[0166] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.< / status>
Claims
1. An interactive information processing method, characterized in that, The method includes: In response to receiving the problem information, the first search result corresponding to the problem information is obtained; Iteratively execute the following operations: In the Nth iteration, determine whether the existing search results lack the necessary information to answer the question; N≥1, and N is an integer; If the existing search results lack the necessary information to answer the question, obtain corresponding supplementary search results for the missing necessary information, and proceed to the next iteration; or, If the current search results do not lack the necessary information to answer the question, the iteration ends; wherein, the search results in the first iteration are the first search results; when N≥2, the search results in the Nth iteration include the search results in the previous iteration and the supplementary search results obtained in the previous iteration; Generate and output a response to the question based on the existing search results.
2. The method according to claim 1, characterized in that, The method includes: In the first iteration, it was determined that the first search result lacked the necessary information to answer the question. In response to the lack of necessary information in the first search result, a first supplementary search result is obtained for the first necessary information missing in the first search result, and a second round of iteration is performed; In the second iteration, it is determined that the first existing search results do not lack the necessary information to answer the question information, and the first existing search results include the first search result and the first supplementary search result; If the first existing search result does not lack the necessary information, the iteration ends, and a response to the question is generated based on the first existing search result and the response is output.
3. The method according to claim 1, characterized in that, The method includes: In the first iteration, it was determined that the first search result lacked the necessary information to answer the question. In response to the lack of necessary information in the first search result, a first supplementary search result is obtained for the first necessary information missing in the first search result, and a second round of iteration is performed; In the second iteration, it is determined that the first existing search results lack the necessary information required to answer the question information. The first existing search results include the first search result and the first supplementary search result. In response to the lack of necessary information in the first existing search results, a second supplementary search result is obtained for the second necessary information that is missing in the first existing search results, and the third iteration is initiated. In the third iteration, it is determined that the second existing search results do not lack the necessary information to answer the question; the second existing search results include the first existing search results and the second supplementary search results; If the second existing search result does not lack the necessary information to reply to the question information, the iteration ends; based on the second existing search result, a reply to the question information is generated and output.
4. The method according to any one of claims 1-3, characterized in that, Determine whether the existing search results lack the necessary information to answer the question, including: Generate a prompt; the prompt includes the question information, the existing search results, and the task instruction; the task instruction is used to prompt the large language model to output at least the status information based on the question information and the existing search results; The prompt is input into a large language model, which then outputs at least status information. This status information indicates whether the existing search results lack the necessary information to respond to the question.
5. The method according to claim 4, characterized in that, The task instruction is also used to prompt the large language model to output missing information; the missing information is the necessary information that the large language model predicts is missing from the current existing search results relative to the necessary information required to reply to the question information. The prompt is input into a large language model, which then outputs at least the following state information: The prompt is input into a large language model, which then outputs the status information and the missing information.
6. The method according to claim 5, characterized in that, The task instruction is also used to prompt the large language model to output valid information, which is necessary information extracted from the existing search results; The large language model outputs at least state information, including: The large language model outputs state information, missing information, and valid information.
7. The method according to claim 6, characterized in that, The prompt also includes information specifying the data format; The large language model outputs state information, missing information, and valid information, including: The large language model outputs the state information, the missing information, and the valid information in a specified data format.
8. The method according to any one of claims 1-3 and 5-7, characterized in that, Generate and output response information to the question based on the existing search results, including: Generate a complete prompt; the complete prompt includes the current existing search results and the question information; The complete prompt is input into the large language model, and the response information returned by the large language model is received.
9. An electronic device, characterized in that, include: Memory, used to store computer program products; A processor for executing a computer program product stored in the memory, wherein when the computer program product is executed, it implements the method described in any one of claims 1-8.
10. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-8.