Intelligent question and answer method and device and computing device cluster

Abstract

The intelligent question and answer method comprises the following steps: performing retrieval based on a first user query of a current round of dialogue to obtain first retrieval knowledge required in the current round of dialogue; processing the first retrieval knowledge and the first user query to obtain a response text; based on the first retrieval knowledge, the response text and feedback of the user to the response text, performing reflection evaluation on the first retrieval knowledge to obtain a reflection evaluation conclusion, the reflection evaluation conclusion being used for indicating the quality of a reference fragment in the first retrieval knowledge, and the reference fragment being a fragment referenced by the response text; when the quality of the reference fragment does not meet the requirement, second retrieval knowledge needed in the next round of dialogue is optimized on the basis of the reference fragment, and the second retrieval knowledge is obtained by conducting retrieval on the basis of the first reference fragment and second user query in the next round of dialogue. According to the method, wrong knowledge in the retrieved knowledge can be corrected in time, the probability of introducing the wrong knowledge is reduced, and the answer quality of the neural network model is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to an intelligent question-answering method, device, and computing device cluster. Background Technology

[0002] With the rapid development of large language models (LLMs), a large number of chatbots have emerged in the industry, such as the chat generative pre-trained transformer (ChatGPT). These chatbots, based on LLMs, can perform complex tasks such as question-and-answer, copywriting, and code generation through dialogue with users. Although LLMs demonstrate strong language understanding and summarization capabilities, their training data is usually general-domain internet data, lacking the latest knowledge and private domain data specific to vertical fields. Therefore, LLMs often exhibit "illusions" when facing specific domains or professional problems.

[0003] To improve the accuracy, timeliness, and professionalism of LLM (Limited Learning Model), the current mainstream approach is retrieval augmented generation (RAG) technology. RAG first builds a knowledge document library. When a user asks a query, it retrieves the most relevant knowledge text from the library and inputs both the user's query and the retrieved knowledge text into the LLM, which then generates a response and returns it to the user. In this process, RAG provides the LLM with additional knowledge information, helping it better understand the query and create a response. By providing relevant knowledge text, RAG can introduce external expertise into the LLM, assisting it in generating higher-quality answers. However, if RAG provides incorrect or irrelevant external knowledge or information, it not only fails to improve the quality of the LLM's answer but may also exacerbate the illusion effect. Therefore, how to promptly correct and optimize poorly performing retrieval knowledge during multi-turn dialogues, reducing the introduction of erroneous knowledge to ensure the LLM can generate accurate answers, is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] This application provides an intelligent question-answering method, apparatus, computing device cluster, computer storage medium, and computer product that can promptly correct and optimize poor-quality retrieval knowledge during multi-turn dialogues, reduce the introduction of erroneous knowledge, and ensure that LLM can generate accurate answers.

[0005] In a first aspect, this application provides an intelligent question-answering method, comprising: performing a retrieval based on a first user query in the latest round of dialogue in a first session to obtain first retrieval knowledge required in the latest round of dialogue; processing the first retrieval knowledge and the first user query through a neural network model to obtain a first response text; performing a reflective evaluation on the first retrieval knowledge based on the first retrieval knowledge, the first response text, and the user's first feedback on the first response text to obtain a first reflective evaluation conclusion, the first reflective evaluation conclusion being used to indicate the quality of a first cited fragment in the first retrieval knowledge, the first cited fragment being a fragment cited by the first response text; and optimizing the second retrieval knowledge required in the next round of dialogue based on the first cited fragment if the quality of the first cited fragment does not meet the requirements, wherein the second retrieval knowledge is obtained by retrieving based on the first cited fragment and a second user query in the next round of dialogue.

[0006] In this way, during the intelligent question answering process, the quality of the retrieved knowledge generated in each round of dialogue is evaluated, and the poor-quality retrieved knowledge is corrected in the next round of dialogue. This can correct the erroneous knowledge in the retrieved knowledge in a timely manner, reduce the probability of introducing erroneous knowledge, and improve the answer quality of the neural network model.

[0007] In one possible implementation, the latest round of dialogue is a non-first round of dialogue within the first session. In this case, a retrieval is performed based on the first user query in the latest round of dialogue to obtain the first retrieval knowledge required for the latest round of dialogue. This includes: retrieving knowledge related to the user's query intent from a knowledge base, where the knowledge base stores knowledge documents and / or knowledge fragments; and filtering the first retrieval knowledge from the knowledge related to the user's query intent based on a second reference fragment. The first retrieval knowledge is weakly correlated with the second reference fragment, or strongly correlated with the second reference fragment and has a higher content richness than the second reference fragment. The second reference fragment is a fragment of the retrieval knowledge required in the previous round of dialogue that was referenced by the response text in the previous round of dialogue, and the quality of the second reference fragment does not meet the requirements. This allows for the removal of erroneous knowledge during the generation of retrieval knowledge in this round, enabling timely correction of the retrieval knowledge; it also allows for timely enrichment and optimization of the retrieval knowledge during the generation of retrieval knowledge in this round.

[0008] In one possible implementation, if the quality type of the second cited fragment is incorrect, the first retrieved knowledge is weakly correlated with the second cited fragment. This corrects the ineffective information gain of the retrieved knowledge. Conversely, if the quality type of the second cited fragment is insufficient content richness, the first retrieved knowledge is strongly correlated with the second cited fragment and has higher content richness than the second cited fragment. This optimizes the effective information gain of the retrieved knowledge.

[0009] In one possible implementation, the method further includes, if the quality of the first reference fragment meets the requirements, performing one or more of the following: storing the first retrieval knowledge; generating a knowledge summary for the first session; or ending the first session. Storing the first retrieval knowledge facilitates quickly finding accurate answers when encountering similar questions later, improving retrieval efficiency. Generating a knowledge summary for the current session helps users understand the content of the session, enhancing the user experience. Furthermore, ending the first session allows for starting a new session.

[0010] In one possible implementation, based on the first retrieved knowledge, the first response text, and the user's first feedback to the first response text, a reflective evaluation of the first retrieved knowledge is performed to obtain a first reflective evaluation conclusion. This includes: calculating co-occurring segments between the first retrieved knowledge and the first response text to obtain first cited segments; and processing the first cited segments and the first feedback through a reflective evaluation model to obtain the first reflective evaluation conclusion. Since the response text is obtained based on the retrieved knowledge, and the response text may cite some segments from the retrieved knowledge, the quality of the retrieved knowledge can be accurately evaluated by evaluating the segments cited in the response text.

[0011] In one possible implementation, the first feedback is the user's follow-up question in the next round of conversation.

[0012] Secondly, this application provides an intelligent question-answering device comprising a retrieval module and a processing module. The retrieval module performs a retrieval based on a first user query in the latest round of dialogue within a first session to obtain first retrieval knowledge required for the latest round of dialogue. The processing module processes the first retrieval knowledge and the first user query using a neural network model to obtain first response text. The processing module is further configured to perform a reflective evaluation of the first retrieval knowledge based on the first retrieval knowledge, the first response text, and the user's first feedback to the first response text, to obtain a first reflective evaluation conclusion. The first reflective evaluation conclusion indicates the quality of a first cited fragment in the first retrieval knowledge, where the first cited fragment is the fragment cited by the first response text. The processing module is further configured to optimize second retrieval knowledge required for the next round of dialogue based on the first cited fragment if the quality of the first cited fragment does not meet the requirements. The second retrieval knowledge is obtained by retrieving the first cited fragment and a second user query in the next round of dialogue.

[0013] In one possible implementation, the latest round of dialogue is a non-first round of dialogue in the current session. In this case, when the retrieval module performs a search based on the first user query in the latest round of dialogue to obtain the first retrieval knowledge required for the latest round of dialogue, it specifically performs the following: Based on the first user query, it searches the knowledge base to obtain knowledge related to the user's query intent, wherein the knowledge base stores knowledge documents and / or knowledge fragments; based on the second reference fragment, it filters the first retrieval knowledge from the knowledge related to the user's query intent, where the first retrieval knowledge is weakly correlated with the second reference fragment, or, the first retrieval knowledge is strongly correlated with the second reference fragment and has a higher content richness than the second reference fragment. The second reference fragment is a fragment of the retrieval knowledge required in the previous round of dialogue that was referenced by the response text in the previous round of dialogue, and the quality of the second reference fragment does not meet the requirements.

[0014] In one possible implementation, if the quality type of the second reference fragment is incorrect, the first retrieved knowledge is weakly correlated with the second reference fragment; if the quality type of the second reference fragment is lacking in content richness, the first retrieved knowledge is strongly correlated with the second reference fragment and its content richness is higher than that of the second reference fragment.

[0015] In one possible implementation, the processing module is further configured to: if the quality of the first reference fragment meets the requirements, perform one or more of the following: store the first retrieval knowledge; generate a knowledge summary of the first session; or, terminate the first session.

[0016] In one possible implementation, when the processing module performs a reflective evaluation on the first retrieval knowledge based on the first retrieval knowledge, the first response text, and the user's first feedback on the first response text to obtain the first reflective evaluation conclusion, it specifically performs the following: calculates the co-occurrence fragments between the first retrieval knowledge and the first response text to obtain the first cited fragments; and processes the first cited fragments and the first feedback through the reflective evaluation model to obtain the first reflective evaluation conclusion.

[0017] In one possible implementation, the first feedback is the user's follow-up question in the next round of conversation.

[0018] Thirdly, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect.

[0019] Fourthly, this application provides a computer-readable storage medium including computer program instructions, which, when executed by a computing device, perform the method described in the first aspect or any possible implementation thereof; or, when executed by a cluster of computing devices, the cluster of computing devices performs the method described in the first aspect or any possible implementation thereof. Exemplarily, the cluster of computing devices may include one or more computing devices.

[0020] Fifthly, this application provides a computer program product containing instructions that, when executed by a computing device, cause the computing device to perform the method described in the first aspect or any possible implementation thereof; or, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in the first aspect or any possible implementation thereof. Exemplarily, a cluster of computing devices may include one or more computing devices.

[0021] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the architecture of an intelligent question-answering system provided in an embodiment of this application;

[0023] Figure 2 This is a schematic diagram illustrating the technical concept of intelligent question answering provided in an embodiment of this application;

[0024] Figure 3 This is a schematic diagram of the architecture of an intelligent question-answering system provided in an embodiment of this application;

[0025] Figure 4 This is a schematic diagram illustrating a process for obtaining a reflective evaluation conclusion provided in an embodiment of this application;

[0026] Figure 5 This is a hardware and software schematic diagram of an intelligent question-answering system provided in an embodiment of this application;

[0027] Figure 6 This is a schematic diagram of the workflow of an intelligent question-answering system provided in an embodiment of this application;

[0028] Figure 7 This is a schematic diagram illustrating the product implementation of a software module in an intelligent question-answering system provided in this application embodiment;

[0029] Figure 8 This is a schematic diagram illustrating the flow and operation of data within an intelligent question-and-answer system, as provided in an embodiment of this application.

[0030] Figure 9 This is a schematic diagram of the interface provided by a client during a single session, as provided in an embodiment of this application.

[0031] Figure 10 This is a flowchart illustrating an intelligent question-answering method provided in an embodiment of this application;

[0032] Figure 11 This is a schematic diagram of the structure of an intelligent question-answering device provided in an embodiment of this application;

[0033] Figure 12 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0034] Figure 13 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0035] Figure 14 This is a schematic diagram of another computing device cluster structure provided in an embodiment of this application. Detailed Implementation

[0036] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0037] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0038] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0039] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0040] First, the relevant technical terms involved in the technical solution provided in this application will be introduced.

[0041] (1) Retrieval Augmented Generation (RAG)

[0042] RAG is a technique that combines retrieval and generation to improve the performance of LLM in information retrieval and text generation tasks. RAG enhances the generative model's capabilities by introducing a retrieval mechanism, providing externally retrieved content before the large model generates an answer, thereby improving the accuracy and relevance of the generated text.

[0043] (2) Hallucination

[0044] Illusion refers to false or erroneous information generated by LLM when generating text. This information does not conform to the facts or is inconsistent with knowledge of the real world.

[0045] (3) Dynamic block partitioning

[0046] In RAG technology, chunking is a crucial step used to divide large text or documents into smaller, more manageable chunks. A document can be very large; splitting it into smaller chunks can improve retrieval efficiency and generation quality, meeting the limitations of large model input lengths. Related technologies typically use offline chunking: documents in a prepared knowledge base are chunked offline, usually based on the text length of the chunks. Once the document is divided into chunks, the content of each chunk remains unchanged. This application uses dynamic chunking technology, where the document is not pre-divided into smaller chunks offline, but rather the chunking result is dynamically determined during the online process based on the dialogue scenario.

[0047] (4) Reflection and Evaluation

[0048] Reflective evaluation typically refers to assessing the results generated by the LLM (Low-Level Machine) to determine whether the generated answers satisfy the user's query. In this application, it mainly refers to evaluating the retrieval knowledge obtained through RAG (Retrieved Information Aggregator) technology to assess whether the retrieval knowledge contributes to the generated content of the LLM and whether it meets the user's query needs.

[0049] The technical solution provided in this application will be described below.

[0050] For example, Figure 1 A schematic diagram of the architecture of an intelligent question-answering system provided in an embodiment of this application is shown. Figure 1As shown, the intelligent question-answering system may include: a knowledge base construction module 110, a query understanding module 120, a retrieval module 130, a prompt generation module 140, an LLM 150, and a response evaluation module 160. The knowledge base construction module 110 is mainly used to segment prepared knowledge documents offline, thus building a knowledge base in advance for use by the retrieval module 130. The knowledge base stores knowledge fragments extracted from the knowledge documents. The query understanding module 120 is mainly used to understand user queries in historical question-answering sessions and the current dialogue, including rewriting and expanding them to transform the original user query into optimized query text. The retrieval module 130 is mainly used to retrieve the knowledge fragments most relevant to the user's query intent from the knowledge base to obtain the corresponding retrieval knowledge. The prompt generation module 140 is mainly used to assemble the retrieval knowledge and the query text output by the query understanding module 120 into a prompt, which is then input into the LLM 150. LLM150 is primarily used to perform inference based on the Prompt from Prompt generation module 140 to generate a response. Response evaluation module 160 is primarily used to evaluate the response results of LLM150, determine whether they meet the user's query requirements, and optimize the response results to obtain the final response content.

[0051] Figure 1 The intelligent question-answering system shown can effectively integrate and retrieve external knowledge bases, addressing the issue of LLMs lacking professional knowledge in answering questions. However, when evaluating the quality of the answers generated by LLMs, this system can only assess the final effectiveness of the LLM's responses and cannot directly reflect the quality of external knowledge retrieved by RAG. Therefore, when using... Figure 1 The intelligent question system shown cannot correct poor-quality retrieved knowledge in a timely manner during multi-round dialogues, making it difficult to avoid introducing erroneous knowledge.

[0052] In view of this, this application provides an intelligent question answering method that can promptly correct and optimize poor-quality search knowledge during the question answering process using LLM, reducing the probability of introducing incorrect knowledge and improving the answer quality of LLM.

[0053] For example, Figure 2 This diagram illustrates a technical concept of intelligent question answering provided in an embodiment of this application. Under this technical concept, the quality of the retrieved knowledge generated in the current round can be directly evaluated. If the quality of the retrieved knowledge generated in this round does not meet the requirements, errors in the retrieved knowledge can be corrected in a timely manner in the next round of dialogue. For example... Figure 2 As shown, the technical concept of this intelligent question answering mainly includes four parts: retrieval, dynamic segmentation, response generation, and reflection and evaluation.

[0054] The retrieval section primarily retrieves knowledge documents and / or knowledge fragments matching the user input from the knowledge base during each round of dialogue in a session. The dynamic chunking section extracts sentences that meet the requirements from the retrieved knowledge documents and / or knowledge fragments during each round of dialogue, and assembles these extracted sentences into text chunks as the retrieved knowledge. The response generation section assembles the user input and generated retrieved knowledge from each round of dialogue into a Prompt and transmits it to the LLM to generate a response (i.e., obtain the LLM-generated content). The reflection and evaluation section reflects on and evaluates the quality of the retrieved knowledge and the LLM-generated content based on the retrieved knowledge generated in each round of dialogue and the user's feedback on the LLM-generated content during the corresponding rounds of dialogue (i.e., user feedback), to obtain a reflection and evaluation conclusion. The reflection and evaluation conclusion can be used to indicate the quality of the fragments in the retrieved knowledge referenced by the LLM-generated content. In this embodiment, a session consists of at least one round of dialogue, and each round of dialogue can be a complete question-and-answer session. For example, a user asks, "What will the weather be like tomorrow?", and the system replies, "Tomorrow is expected to be sunny turning cloudy." The user then asks, "And the temperature?", and the system adds, "The high will be around 25 degrees Celsius." This can be understood as one conversation, and a user's question and the system's response can be understood as one round of dialogue. Furthermore, a single conversation window can contain multiple conversations, each covering different topics. For instance, within a conversation window, a user might first engage in a weather-related conversation, then a shopping-related conversation. In this case, the weather-related conversation can be considered one conversation, and the shopping-related conversation another. Additionally, within a conversation window, when the reflective evaluation conclusion generated by a round of dialogue indicates that the quality of the fragments referenced by the LLM-generated content in the retrieved knowledge meets the requirements, a conversation can also be considered complete. For example, if a session window contains 15 rounds of dialogue, and the reflective evaluation conclusions generated from rounds 1 to 9 all indicate that the quality of the fragments in the retrieved knowledge referenced by the LLM-generated content does not meet the requirements, while the reflective evaluation conclusion generated from round 10 indicates that the quality of the fragments in the retrieved knowledge referenced by the LLM-generated content meets the requirements, then rounds 1 to 10 are considered one session, and round 11 is considered a new session, being the first round of the latest session. In this case, the topics covered in rounds 1 to 10 can be the same as or different from those covered in round 11. Furthermore, sessions within different session windows belong to different sessions.

[0055] exist Figure 2Under the illustrated technical concept, when the quality of the fragments referenced by the LLM-generated content in the retrieved knowledge, as indicated by the reflective evaluation conclusion obtained in the current round of dialogue, meets the requirements, it indicates that correct retrieved knowledge and LLM-generated content have been obtained, and thus the current session can end. When the reflective evaluation conclusion obtained in the current round of dialogue indicates that the quality of the fragments referenced by the LLM-generated content in the retrieved knowledge does not meet the requirements, it indicates that there are fragments in the retrieved knowledge that do not meet the quality requirements, which introduces erroneous knowledge. To correct this erroneous knowledge, a next round of dialogue can be initiated. When correcting the fragments in the retrieved knowledge generated in the previous round that do not meet the quality requirements, in the dynamic chunking part of this round, sentences weakly related to the fragments in the retrieved knowledge generated in the previous round can be extracted from the knowledge documents and / or knowledge fragments retrieved in this round as retrieved knowledge, thereby correcting the ineffective information gain of the retrieved knowledge. In this way, during the dynamic chunking process, the reflective evaluation conclusion of the previous round can guide the timely correction of poor-quality retrieved knowledge and the generation of higher-quality retrieved knowledge.

[0056] Continue reading Figure 2 Taking a two-round dialogue as an example, in the first round, during the retrieval process, the query text obtained by rewriting or expanding the user's original query through query understanding can be used to retrieve the N knowledge documents (and / or knowledge fragments) with the highest relevance scores to the query text. Since there are no reflective evaluation conclusions for reference in the first round of dialogue, algorithms such as semantic relevance can be used to calculate the semantic relevance score of each sentence in the query text and the N knowledge documents (and / or knowledge fragments), and the top K sentences with the highest scores are selected as the text content that meets the conditions, forming the final retrieval knowledge. Next, the retrieval knowledge and query text can be assembled into a Prompt, input into the LLM, and a response text can be generated, thus obtaining the LLM generated content. Finally, based on the retrieval knowledge retrieved in this round, the LLM generated content (i.e., the generated response), and the user's feedback on the LLM generated content in this round (i.e., user feedback), the quality of the retrieval knowledge and the LLM generated content can be reflected upon and evaluated. In the reflection and evaluation process, a citation source detection can be performed first to identify fragments in the retrieved knowledge that are referenced by the LLM-generated content (hereinafter referred to as "citation fragments"). Then, a citation source reflection and evaluation is performed to determine the quality of the citation fragments, thereby judging whether the referenced retrieved knowledge meets the user's query requirements. The reflection and evaluation conclusion is recorded as the basis for the next round of dynamic segmentation optimization. Suppose that the reflection and evaluation conclusion of this round indicates that the quality of the fragments in the retrieved knowledge referenced by the LLM-generated content does not meet the requirements, then the next round of dialogue can proceed.

[0057] In the second round of dialogue, during the retrieval process, the query text obtained by rewriting or expanding the user's original query through query understanding can be used to retrieve the N knowledge documents (and / or knowledge fragments) with the highest relevance scores to the query text from the knowledge base. Since there are reflective evaluation conclusions to refer to in this round of dialogue, these conclusions can be combined with those from the previous round to extract text content weakly related to the poor-quality citation fragments indicated by the previous round's reflection evaluation conclusions. This extracted text content can then be used to form the final retrieval knowledge, thereby correcting the ineffective information gain of the retrieval knowledge.

[0058] After obtaining the retrieved knowledge during the second round of dialogue, the retrieved knowledge and query text can be assembled into a Prompt, input into the LLM, and a response text can be generated, thus obtaining the LLM-generated content. Finally, based on the retrieved knowledge, the LLM-generated content, and user feedback on the LLM-generated content (i.e., user feedback), the quality of the retrieved knowledge and the LLM-generated content can be re-evaluated. Assuming that the re-evaluation conclusion of this round indicates that the quality of the fragments in the retrieved knowledge referenced by the LLM-generated content meets the requirements, the session can then be terminated.

[0059] exist Figure 2 Under the technical concept of intelligent question answering, the knowledge retrieved in a round of dialogue is extracted from the retrieved knowledge documents (and / or knowledge fragments). During the extraction process, the reflection and evaluation conclusions of the previous round of retrieved knowledge can guide the process, revealing where the retrieved knowledge went wrong. This allows the system to consider data that differs from the previous round's retrieved knowledge in the current round, enabling timely correction of poor-quality retrieved knowledge, improving the quality of the retrieved knowledge extracted in the current round, significantly enhancing the LLM's answering capabilities, and reducing the number of rounds required for user question answering. Furthermore, because the reflection and evaluation process considers the quality of both the retrieved knowledge and the LLM's response results, and uses user feedback as the evaluation basis rather than solely relying on the LLM's capabilities, the reflection and evaluation conclusions directly reflect the quality of external knowledge retrieved by RAG. Therefore, the reflection and evaluation conclusions of the previous round of retrieved knowledge can be used as the optimization basis for dynamic segmentation in the current round.

[0060] The following describes the specific implementation process of the above technical concept.

[0061] For example, Figure 3 A schematic diagram of the architecture of an intelligent question-answering system provided in an embodiment of this application is shown. Figure 3 As shown, the intelligent question-answering system 300 may include: a client 310 and a server 320.

[0062] Client 310 can be a desktop application, mobile application, web application, or web-based application, and can be deployed on electronic devices such as mobile phones and computers. Client 310 can provide a user interface (UI) to allow users to interact with it, such as inputting their desired query or viewing content generated by the intelligent question-answering system 300. The user interface can be a graphical user interface (GUI), a command-line interface (CLI), or other forms of interactive interface; no limitation is made here. In this embodiment, client 310 can transmit the user's query (i.e., user inquiry) to server 320 and display the LLM-generated content returned by server 320 to the user. In some embodiments, client 310 can also display guidance information when displaying the LLM-generated content returned by server 320. This guidance information is used to guide the user's evaluation and feedback on the retrieved knowledge and / or LLM-generated content in this round. Users can selectively use the guidance information as the query text for the next round of dialogue, or they can use other queries to ask follow-up questions; there are no restrictions here.

[0063] The server 320 may be, but is not limited to, a device or platform that can provide at least computing resources, such as a server or cloud server. The server 320 may be configured with, but is not limited to, a knowledge base construction module 321, a knowledge base 322, a query understanding module 323, a retrieval module 324, a dynamic chunking module 325, a Prompt generation module 326, an LLM 327, and a reflection and evaluation module 328.

[0064] The knowledge base construction module 321 is primarily used to pre-build the knowledge base 322 in the offline phase for use by the retrieval module 324. During the knowledge base construction process, the knowledge base construction module 321 can retain the original content of each knowledge document without pre-segmenting it. Instead, it directly stores each knowledge document in a knowledge base such as an inverted index or vector index library, meaning the original knowledge documents are stored at the document level. For example, knowledge documents can be in various formats such as PDF, PPTX, Docx, txt, HTML, and Markdown. Alternatively, the knowledge base construction module 321 can segment the prepared knowledge documents offline and store each text segment in a knowledge base such as an inverted index or vector index library. Or, in the offline environment, it can store some knowledge documents at the document level while segmenting other knowledge documents before storing them in the knowledge base. The specific approach depends on the actual situation and is not limited here. In some embodiments, considering that a text fragment after being segmented usually contains only a partial content of a knowledge document, resulting in the loss of article content integrity and topic consistency, which increases the difficulty of retrieving the correct knowledge, it is advisable to prioritize storing knowledge documents at the document granularity in order to preserve the complete content of the knowledge document, avoid losing article content integrity and topic consistency, and reduce the difficulty of retrieving the correct knowledge.

[0065] Knowledge Base 322 is primarily used to store knowledge documents at the document and / or text block level using inverted indexes or vector indexes.

[0066] The query understanding module 323 is primarily used to rewrite or expand the user's original query (i.e., the query from client 310) in the current round, rewriting the original query into optimized query text to make the content more fluent. For example, when the user queries "How's the weather?", and the user allows the system to obtain their current location, the query understanding module 323 can optimize the user's query to "How's the weather in Beijing today?". For instance, the query understanding module 323 can combine historical queries and historical answers to optimize the original query in the current round, supplementing necessary information mentioned in the context, eliminating ambiguity, and making the content more fluent.

[0067] The retrieval module 324 is primarily used to retrieve the N knowledge documents (and / or knowledge fragments) with the highest relevance scores to the user's query from the knowledge base 322. For ease of description, "knowledge documents and / or knowledge fragments" will be collectively referred to as "knowledge data" below. For example, the retrieval module 324 may, but is not limited to, use algorithms such as cosine similarity to calculate the similarity between the optimized query text and the knowledge data stored in the knowledge base 322; then, the N knowledge data with the highest similarity will be used as the data retrieved in this instance.

[0068] The dynamic segmentation module 325 is mainly used to dynamically segment the N knowledge data retrieved by the retrieval module 324 to form the final retrieval knowledge. In this embodiment, during the first round of dialogue in a session, the dynamic segmentation module 325 can use algorithms such as semantic relevance to calculate the semantic relevance score of each sentence in the query text and the N knowledge data, and select the Top K sentences with the highest scores as the text content that meets the conditions, and form them into text segments as retrieval knowledge. During non-first rounds of dialogue in a session, the dynamic segmentation module 325 can use algorithms such as semantic relevance to calculate the semantic relevance score of each sentence in the query text and the N knowledge data, and select the Top M sentences with the highest scores. Then, the dynamic segmentation module 325 combines the reflection and evaluation conclusions of the retrieval knowledge generated in the previous round of dialogue provided by the reflection and evaluation module 328, and selects the Top K sentences (or other numbers, not limited here) that meet the conditions from the Top M sentences, and uses them as retrieval knowledge, thereby correcting the retrieval knowledge of poor quality in a timely manner. In some embodiments, when the quality of a fragment referenced by the LLM-generated content in the retrieved knowledge does not meet the requirements, the corresponding referenced fragment is not necessarily completely wrong; it may be insufficiently detailed. Therefore, if all fragments that do not meet the quality requirements from the previous round of retrieved knowledge are removed when extracting retrieved knowledge from candidate sentences, it may be an overcorrection, potentially leading to problems with the quality of the retrieved knowledge generated in the current round. Considering the above, the quality type of the fragments referenced by the LLM-generated content in the retrieved knowledge can be added to the reflection and evaluation conclusion. This allows the data extraction method to be selected based on the quality type when the quality of the fragments referenced by the LLM-generated content in the retrieved knowledge does not meet the requirements. In this case, the reflection and evaluation conclusion can indicate not only the quality of the fragments referenced by the LLM-generated content in the retrieved knowledge but also the quality type of the fragments referenced by the LLM-generated content in the retrieved knowledge. The quality type can include: error or insufficient detail, etc. When the quality type is erroneous, the information entropy gain algorithm can be used for retrieval knowledge generation. This involves calculating the information gain score of each character (token) between the selected candidate sentences (the Top M sentences) and the cited fragments from the previous round of retrieval knowledge that do not meet the quality requirements, and retaining the Top K sentences with the highest scores. This process extracts sentences from the candidate sentences that have low relevance to the erroneous cited fragments recorded in the previous round of retrieval knowledge, thereby correcting the ineffective information gain of the retrieval knowledge. When the quality type is incomplete, the information completeness algorithm can be used for retrieval knowledge generation. This involves calculating the semantic relevance score of each sentence in the selected candidate sentences and the cited fragments from the previous round of retrieval knowledge that do not meet the quality requirements, and retaining the Top K sentences with the highest scores, excluding the cited fragments from the previous round of retrieval knowledge that do not meet the quality requirements.This process extracts sentences from candidate sentences that are highly relevant to the less detailed references recorded in the previous round of retrieval knowledge, but with more detailed content. In other words, it extracts sentences that are strongly related to the less detailed references recorded in the previous round of retrieval knowledge and have richer content, thereby optimizing the effective information gain of the retrieval knowledge in this round. Additionally, the quality type can also be "Correct," indicating that the quality of the fragments referenced by the LLM-generated content in the retrieval knowledge meets the requirements.

[0069] The Prompt generation module 326 is mainly used to assemble the retrieval knowledge output by the dynamic chunking module 325 and the query text output by the query understanding module 323 into a Prompt, which is then input into the LLM 327. For example, the Prompt generation module 326 can add the retrieval knowledge and query text to a preset Prompt template to obtain the Prompt.

[0070] The LLM327 is mainly used to perform inference based on the Prompt generated by the Prompt generation module 326 to generate response text, that is, to obtain generated content related to the query text (also known as "inference content" or "LLM generated content").

[0071] The reflection and evaluation module 328 is primarily used to reflect on and evaluate the quality of the retrieved knowledge and LLM-generated content based on the retrieved knowledge, LLM-generated content, and user feedback (i.e., user feedback) in this round of retrieval. This evaluation forms the basis for optimizing retrieval knowledge and guides the generation of retrieval knowledge in the next round. Specifically, such as... Figure 4As shown, in S401, a fuzzy matching algorithm is used to process the retrieved knowledge and LLM-generated content to identify fragments (also called "reference fragments") in the retrieved knowledge that are referenced by the LLM-generated content. For example, co-occurring fragments between the retrieved knowledge and the LLM-generated content can be calculated; when the length of the co-occurring characters exceeds a set threshold, the fragment is considered a reference fragment. In S402, the reference fragments are evaluated based on user feedback to the LLM-generated content (i.e., user feedback) to obtain a reflective evaluation conclusion. For example, a pre-trained reflective evaluation classification model can be used to process the reference fragments and user feedback to obtain a reflective evaluation conclusion for the retrieved knowledge. This reflective evaluation conclusion can indicate the quality of fragments referenced by the LLM-generated content in the current round of retrieved knowledge. Of course, the reflective evaluation conclusion can also indicate the quality type of fragments referenced by the LLM-generated content in the current round of retrieved knowledge, such as: erroneous, incomplete, or correct. As a possible implementation, the format of the reflective evaluation conclusion can be, but is not limited to: <reflective evaluation signal value sign, reference fragment text ref>. The reflection and evaluation signal value can include any one of the following: incorrect retrieved knowledge, insufficiently detailed retrieved knowledge, and correct retrieved knowledge. In some embodiments, when the reflection and evaluation conclusion indicates that the quality of the fragments referenced by the LLM-generated content in this round of retrieved knowledge meets the requirements (i.e., the retrieved knowledge in this round is correct), it indicates that the retrieved knowledge in this round meets the user's query requirements. Therefore, the current session can be ended, and the retrieved knowledge in this round can be stored as a new knowledge document in the knowledge base 322, i.e., retrieval knowledge sedimentation, so that when encountering similar queries, the knowledge document that meets the requirements can be quickly retrieved from the knowledge base, improving retrieval efficiency. Because the dynamic chunking module 325 will further dynamically extract the top K sentences from the retrieved candidate knowledge data and assemble them into retrieval knowledge based on the user query and the reflection and evaluation conclusion, the retrieval knowledge stored in the knowledge base may be different from the original knowledge data content. Therefore, storing the retrieval knowledge that meets the requirements in the knowledge base can achieve the purpose of sedimenting useful retrieval knowledge during the dialogue. Of course, it is also possible to choose not to perform retrieval knowledge sedimentation, depending on the actual situation, which is not limited here. In some embodiments, user feedback may be the query text entered by the user in the next round, or it may be the user's direct feedback on the content generated by the LLM in this round (for example, the user can directly select the quality of the content generated by the LLM in this round), or a combination of both, etc. The specifics may vary depending on the actual situation and are not limited here.

[0072] In some embodiments, the architecture of the intelligent question-answering system 300 described above can also be... Figure 5 As shown in the form. (e.g.) Figure 5As shown, the intelligent question-answering system 300 can be divided into software, storage, and hardware components. The software component mainly consists of a knowledge base construction module 321, a query understanding module 323, a retrieval module 324, a dynamic chunking module 325, a prompt generation module 326, an LLM 327, and a reflection and evaluation module 328. The storage component mainly consists of a knowledge base 322, an inference device, and a training device. The inference device is mainly used for inference with the LLM 327 and is the service device required for online inference. The training device is mainly used for training the LLM 327 and is the service device required for offline training. The hardware component mainly consists of a client 310, a server (or server cluster), and graphics cards. The server (or server cluster) is mainly the physical machine used to deploy the various modules of the system. The graphics cards are mainly the graphics card resources used for training and deploying the models involved in the system (such as the reflection and evaluation module 328).

[0073] The following section uses the example of storing data at the document level in the knowledge base to introduce the workflow of this intelligent question-answering system 300. Figure 6 As shown, the workflow of an intelligent question-answering system may include:

[0074] S601, Knowledge Base Construction. In this step, the original knowledge documents can be stored at the document level in a knowledge base such as an inverted index or a vector index library. First, the user needs to prepare a set of knowledge documents, each knowledge document being a text file. This set of knowledge documents is then stored in an inverted index (such as Elasticsearch) or a vector index (such as Faiss), effectively building the knowledge base index offline. Additionally, when the intelligent question-answering system evaluates the retrieved knowledge and confirms it meets the user's query requirements, the retrieved knowledge can be stored in the knowledge base index.

[0075] S602, Query Understanding. In this step, the user's original query can be rewritten or expanded, transforming it into an optimized query text. Specifically, the system first receives the user's query text, which can be the content of the user's first-round query or a query from a multi-round query during the dialogue. When it is a query from a multi-round query during the dialogue, the user query can be rewritten based on historical questions and answers and the current user query, mainly by supplementing necessary information mentioned in the context, eliminating ambiguity, and making the content more fluent.

[0076] S603, Retrieval. In this step, the N knowledge documents with the highest relevance scores to the optimized query text can be retrieved from the knowledge base.

[0077] S604, Dynamic Blocking. In this step, if there is no reflection and evaluation conclusion from the previous round, a semantic relevance algorithm can be used to generate retrieval knowledge. This involves calculating the vector semantic relevance score (i.e., the vector inner product) of each sentence in the optimized query text and candidate documents, selecting the top K sentences with the highest scores, and forming the retrieval knowledge for this round. If the reflection and evaluation conclusion from the previous round is provided, sentences that meet the conditions in the candidate documents can be extracted based on the optimized query text and the previous round's reflection and evaluation conclusion, and then formed into the retrieval knowledge for this round (see above for details). Figure 3 The description of the dynamic chunking module 325 in the previous section will not be repeated here. In some embodiments, the acceptable input length of the LLM used to generate the response is L, so the total length of the Top K sentences does not exceed L, thereby enabling all retrieved knowledge content to be input into the LLM.

[0078] S605, Generate Response. In this step, the retrieval knowledge from this round and the optimized query text can be assembled into a Prompt, which is then input into the LLM to generate the response text (i.e., the LLM-generated content).

[0079] S606, Display of Response Results and Guidance Information. In this step, the server in the system can send response results and guidance information to the client for user use. Specifically, this includes: LLM generated content, links to referenced search knowledge documents (optional), and guidance information to encourage users to provide feedback and evaluation (optional).

[0080] S607, Reflection and Evaluation. In this step, the quality of the retrieved knowledge and LLM-generated content can be reflected upon and evaluated based on the retrieved knowledge, LLM-generated content, and user feedback (such as the user's follow-up query text and / or the user's feedback on the LLM-generated content). The user's follow-up query can be provided using guidance information or it can be provided independently without guidance information. For the reflection and evaluation process, please refer to the preceding section. Figure 3 The relevant description of the reflection and evaluation module 328 in the text will not be repeated here.

[0081] After introducing the workflow of the intelligent question-answering system 300, the following section will introduce the product implementation of its software modules.

[0082] For example, Figure 7This illustration shows a schematic diagram of the product implementation of a software module in an intelligent question-answering system according to an embodiment of this application. The implementation of the software module in the server 320 of the intelligent question-answering system 300 can be, but is not limited to, program code included in machine learning or deep learning platform software and deployed on server hardware. This program code can, but is not limited to, exist within the runtime engine, memory management, and communication management modules of the platform software, as well as outside existing modules, etc. Furthermore, this program code can also run in the server's host memory and / or graphics processing unit (GPU) memory. Figure 7 As shown, the query understanding module, retrieval module, dynamic segmentation module, knowledge base construction module, reflection and evaluation module, and response generation module in the intelligent question-answering system 300 can all be program code deployed on server hardware. Furthermore, the product implementation of the intelligent question-answering system 300 can include user interaction content such as query entry, result viewing, and guided feedback. Through the query entry, users can enter query text content in the dialogue input box. This content can be the first round of query in a question-and-answer session, or multiple rounds of query (i.e., follow-up questions) during a question-and-answer process. Through result viewing, users can receive the response results provided by the system, view the LLM's answer text content, and the knowledge documents referenced in this round of retrieval, etc. Through guided feedback, users receive both the response results and guided information provided by the system. This guided information guides users to evaluate the referenced retrieval knowledge and the LLM's response results. Users can selectively use the guided text as the query text for the next round of dialogue, or they can use other queries for the next round of follow-up questions. Regardless of whether the user uses guided information text for feedback, they can still engage in dialogue through the dialogue input box.

[0083] The following is based on Figure 2 The technical concept shown describes how data is... Figure 3 The operation of the intelligent question-and-answer system 300 shown is described.

[0084] Please see Figure 8After a user initiates a query, the query can be optimized to obtain the query text. Once the query text is obtained, a search can be performed in the knowledge base to retrieve the required knowledge data (i.e., knowledge documents and / or knowledge fragments) and / or previously stored retrieval knowledge. Then, in the dynamic chunking section, combining the current user query and the reflection and evaluation conclusions of the retrieval knowledge generated in the previous round of dialogue, the Top K sentences that meet the requirements are extracted from the retrieved knowledge data and / or retrieval knowledge and combined to form the retrieval knowledge required for this round of dialogue. Next, using the retrieval knowledge generated in this round, LLM-generated content is obtained and displayed to the user. After the user provides feedback on the LLM-generated content, the retrieval knowledge and LLM-generated content generated in this round of dialogue can be reflected upon and evaluated to obtain a reflection and evaluation conclusion. When the reflection and evaluation conclusion indicates that the quality of the fragments referenced by the LLM-generated content in the retrieval knowledge generated in this round of dialogue meets the requirements, the retrieval knowledge generated in this round of dialogue can be stored in the knowledge base, and the current session can be ended. When the reflection and evaluation conclusion indicates that the quality of fragments referenced by LLM-generated content in the retrieved knowledge generated in this round of dialogue does not meet the requirements, the reflection and evaluation conclusion obtained in this round of dialogue can be used as the basis for optimizing the generation of retrieved knowledge in the next round of dialogue. It should be understood that the retrieval knowledge accumulation in the reflection and evaluation can be configured according to the actual situation, and there are no restrictions here.

[0085] To facilitate understanding of the intelligent question-answering system described above, the following example illustrates how the system operates.

[0086] Please see Figure 9 , Figure 9 This illustrates a schematic diagram of the interface provided by the client during a single session. For example... Figure 9 As shown, after the user enters the query text in the dialog input box of the system interface in each round of dialogue, the system provides an interface with LLM response results, referenced search knowledge links, and guidance information. Figure 9 In this process, four rounds of dialogue occurred between the user and the system. The first round of dialogue used incorrect retrieval knowledge that did not help the LLM generate the correct response. Therefore, the user initiated a second round of dialogue with follow-up questions, and the system promptly optimized the retrieval knowledge based on these follow-up queries. The second round of dialogue used insufficient retrieval knowledge that also did not help the LLM generate the correct response. Therefore, the user initiated a third round of dialogue with follow-up questions, and the system promptly optimized the retrieval knowledge based on these follow-up queries. The third round of dialogue used correct retrieval knowledge that helped the LLM generate a complete response, meeting the user's needs. Therefore, the user confirmed the response, and the system then summarized and displayed the retrieval knowledge that satisfied the user's query and stored it in the knowledge base. The system's operation during these four rounds of dialogue is as follows:

[0087] 1) Query Understanding: The system receives the user's query text. During the first round of dialogue, the query text is input into the open-source LLM, which rewrites the original user query and outputs an optimized query text. For example, if the original query text is "Python code that matches corporate email addresses using regular expressions", the optimized query text can be "Python code that matches corporate email addresses using regular expressions".

[0088] 2) Retrieval: The system retrieves the N knowledge data with the highest relevance scores to the optimized query from the pre-built knowledge base.

[0089] 3) Dynamic Blocking: The relevance score between the optimized query text and each sentence in the candidate knowledge data is calculated sequentially, and the Top K sentences with the highest relevance scores are selected. The total length of the selected sentences does not exceed the input threshold of the LLM used for inference. The first round of retrieval knowledge is derived from the document "Python regular expressions".

[0090] 4) Response generation: The system combines the optimized query text with the retrieval knowledge obtained from the dynamic segmentation step to form a Prompt, which is then input into the LLM used for inference to obtain the response text (i.e., LLM generated content).

[0091] 5) Display of Response Results and Guidance Information: The system sends the generated response text of LLM, the document link of the referenced knowledge "Python regular expressions", and the guidance information text "Was the referenced document and answer helpful? Please continue to ask questions with 'answered' or 'XX is not correct' or 'XX is not detailed enough'" to the client for user use.

[0092] 6) Reflective Evaluation: Based on the retrieved knowledge, the LLM-generated response, and the second-round follow-up query text provided by the guidance information, “The email format is incorrect, this code covers too broad a format,” the quality of the retrieved knowledge and generated response is reflected and evaluated. Specifically, the reflection and evaluation consists of three sub-steps: (1) Reference Source Detection: The reference fragments between the retrieved knowledge and the response results of the large model are found through the fuzzy matching algorithm, that is, the co-occurrence fragments between the retrieved knowledge and the response text are calculated. When the length of the co-occurring characters exceeds the set threshold, the fragment is the reference source. The reference fragment content of the first round of dialogue is “pattern=re.compile(r'^[a-zA-Z0-9._-]+@[a-zA-Z0-9-]+\.[a-zA-Z.]{2,18}$')”. (2) Reference Source Reflective Evaluation: The reflection and evaluation classification model receives two inputs: user feedback “The email format is incorrect, this code covers too broad a format” and the reference fragment, and performs quality judgment on the reference source. The reflection and evaluation model here can be a Chatglm3 model, which is used to obtain the reflection and evaluation classification signal (i.e., the classification result): "The retrieved knowledge is incorrect". The reflection and evaluation signal and the cited source fragment are recorded as the reflection and evaluation conclusion, i.e., <The retrieved knowledge is incorrect, pattern=re.compile(r'^[a-zA-Z0-9._-]+@[a-zA-Z0-9-]+\.[a-zA-Z.]{2,18}$')>.

[0093] 7) Query Understanding: During the second round of dialogue, the query text of this round is concatenated with the query from the first round and the response from the large model, and then input into the open-source LLM. The LLM output rewrites the original user query, producing an optimized query text. For example, the original query "This code covers too broad a format" is optimized to "Python code that matches corporate email addresses using regular expressions".

[0094] 8) Retrieval: Retrieve the N knowledge data with the highest relevance scores to the optimized query text from the knowledge base.

[0095] 9) Dynamic Blocking: Based on the reflection and evaluation conclusion of the first round, sentences that meet the conditions in the candidate knowledge data are extracted. The current reflection and evaluation conclusion is <The retrieved knowledge is incorrect, pattern=re.compile(r'^[a-zA-Z0-9._-]+@[a-zA-Z0-9-]+\.[a-zA-Z.]{2,18}$')>. Then, the system generates retrieved knowledge based on the information entropy gain algorithm. Specifically, firstly, the relevance score between the optimized query and each sentence in the candidate document is calculated sequentially. The top 2*TopK sentences with the highest relevance scores are selected. Then, the information gain score of the character (token) between each sentence and the quoted fragment is calculated sequentially. The top K sentences with the highest scores are retained as dynamic blocks. The total length of the selected sentences does not exceed the input threshold of the large model used for inference. The retrieved knowledge in the second round comes from the document "Regular Expressions (II) Commonly Used Regular Expressions - Verifying Email".

[0096] 10) Generate response: The system concatenates the optimized query text with the retrieval knowledge obtained from the dynamic segmentation step, assembles it into a Prompt, and inputs it into the LLM used for inference to obtain the response text.

[0097] 11) Display of Response Results and Guidance Information: The system sends the generated response text of LLM, the document link of the referenced search knowledge "Regular Expressions (II) Common Regular Expressions - Verifying Emails", and the guidance information text "Was the referenced document and answer helpful? Please continue to ask questions using 'Answered' or 'XX is not correct' or 'XX is not detailed enough', for the user to use."

[0098] 12) Reflection and Evaluation: After the user inputs "The answer is not detailed, what are the rules followed by the beginning of the expression" in the third round of dialogue, the quality of the retrieved knowledge and generated response is reflected and evaluated based on the retrieved knowledge, the LLM generated response, and the user's next follow-up query text "The answer is not detailed, what are the rules followed by the beginning of the expression" provided by the guidance information. Specifically, the reflection and evaluation is divided into 3 sub-steps: (1) Reference source detection: The reference fragment between the retrieved knowledge and the LLM response result is found by using a fuzzy matching algorithm, that is, the co-occurrence fragment between the retrieved knowledge and the response text is calculated. When the length of the co-occurring characters exceeds the set threshold, the fragment is the reference source fragment. The reference fragment content of the second round of dialogue is "letreg=^\w+((-\w+)|(\.\w+))*@[A-Za-z0-9]+((\.|-)[A-Za-z0-9]+)*\.[a-zA-Z0-9]+$ / ". (2) Reflective Evaluation of Citation Sources: The reflective evaluation classification model receives two inputs: user feedback "The answer is not detailed enough, what are the rules followed by the beginning of the expression?" and the cited text fragment, to determine the quality of the citation source. Here, the reflective evaluation model can be a Chatglm3 model, which is used to obtain the reflective evaluation classification signal (i.e., the classification result): "The retrieved knowledge is not detailed enough". Then, the reflective evaluation conclusion is recorded, i.e., <The retrieved knowledge is not detailed enough, let reg=^\w+((-\w+)|(\.\w+))*@[A-Za-z0-9]+((\.|-)[A-Za-z0-9]+)*\.[a-zA-Z0-9]+$ / >.

[0099] 13) Query Understanding: During the third round of dialogue, the query text of this round is concatenated with the query texts from previous rounds and the LLM responses, and then input into the open-source LLM. The LLM output rewrites the original user query, producing an optimized query text. For example, the original query "The answer is not detailed; what are the rules followed by the beginning of the expression?" is optimized to "What are the rules followed by the regular expression for matching Python code for corporate email addresses?"

[0100] 14) Retrieval: Retrieve the N knowledge data with the highest relevance scores to the optimized query text from the knowledge base.

[0101] 15) Dynamic Segmentation: Based on the optimized query text and the conclusion of the second round of reflection and evaluation, sentences are extracted from the candidate knowledge data to form retrieval knowledge. The current reflection and evaluation conclusion is <retrieval knowledge is not detailed, let reg=^\w+((-\w+)|(\.\w+))*@[A-Za-z0-9]+((\.|-)[A-Za-z0-9]+)*\.[a-zA-Z0-9]+$ / >, then the system generates retrieval knowledge based on the content completeness algorithm. Specifically, firstly, the relevance score between the optimized query text and each sentence in the candidate knowledge data is calculated sequentially, and the top 2*TopK sentences with the highest relevance scores are selected. Then, the semantic relevance score between each sentence and the cited fragment is calculated sequentially, and the top K sentences with the highest scores (excluding the cited fragment) are retained as retrieval knowledge. The total length of the selected sentences does not exceed the input threshold of the LLM used for inference. The retrieval knowledge in the third round comes from the document "Regular Expressions (II) Commonly Used Regular Expressions - Verifying Emails".

[0102] 16) Generate response: Combine the optimized query text with the retrieval knowledge obtained from the dynamic chunking step to form a Prompt, which is then input into the LLM to obtain the response text.

[0103] 17) Display of Response Results and Guidance Information: The system sends the LLM generated response text, the document link of the referenced search knowledge "Regular Expressions (II) Common Regular Expressions - Verifying Emails", and the guidance information text "Was the referenced document and answer helpful? Please continue to ask questions using 'Answered' or 'XX is not correct' or 'XX is not detailed enough'" to the client for user use.

[0104] 18) Reflection and evaluation: After the user inputs "End dialogue, generate knowledge summary" in the fourth round of dialogue, the quality of the retrieved knowledge and generated response is reflected and evaluated based on the retrieved knowledge, the LLM generated response, and the user's next round of follow-up query text "End dialogue, generate knowledge summary" provided by the guidance information. Specifically, the reflection and evaluation is divided into 3 sub-steps: (1) Reference source detection: The reference fragment between the retrieved knowledge and the LLM response result is found by using the fuzzy matching algorithm, that is, the co-occurrence fragment between the retrieved knowledge and the response text is calculated. When the length of the co-occurring characters exceeds the set threshold, the fragment is the reference source fragment. The reference fragment content of the third round of dialogue is "Rule analysis 1. The beginning is numbers, letters and underscores (1 to multiple digits) 2. It can also be - numbers, letters and underscores or . numbers, letters and underscores, the whole is zero to multiple\n Summary => The email name is composed of "numbers, letters, underscores, -, .", but - / . cannot appear consecutively or be the beginning\n That is =>\w+((-\w+)|(\.\w+))*". (2) Source Reflection Evaluation: The reflection evaluation classification model receives two inputs: user feedback "End the dialogue, generate a knowledge summary" and a cited fragment. It then performs a quality judgment on the cited source. The reflection evaluation model here can be a Chatglm3 model. Using this model, the reflection evaluation classification signal (i.e., the classification result) is obtained: "The retrieved knowledge is correct". When the reflection evaluation conclusion is that the user's query needs are met, the retrieved knowledge content is stored as a new knowledge document in the knowledge base.

[0105] 19) Display of response results and guidance information: The system sends the retrieved knowledge stored in the knowledge base during the reflection and evaluation steps to the client, specifically including the document link of "Regular Expressions (II) Common Regular Expressions - Verifying Email" and content fragments referenced by LLM, for the user to use.

[0106] As described in the diagram above, in a multi-turn dialogue session, by combining the retrieved knowledge, the LLM-generated content, and the user feedback of the current turn, the quality of the generated retrieved knowledge can be accurately assessed. When the quality of the retrieved knowledge is poor, the quality of the LLM-generated content is likely to be poor as well; therefore, the quality of the LLM-generated content can also be accurately assessed. Furthermore, by incorporating erroneous knowledge from the retrieved knowledge in the current turn and optimizing the retrieved knowledge generation for the next turn, poor-quality retrieved knowledge can be corrected in a timely manner, improving the quality of retrieved knowledge extracted in the next turn, significantly enhancing the LLM's answering ability, and reducing the number of turns required for user question-and-answer sessions.

[0107] It should be understood that, Figure 9 The fourth round of dialogue can be added optionally; no restrictions are set here. Figure 9In the process, users can choose to confirm the correctness of the answer in the third round of dialogue, start a new dialogue directly in the same window, or close the dialogue window, etc. However, it is best to display a summary of the knowledge from the dialogue at the end of each session to improve the user experience. The generation of the dialogue summary can be triggered by the user or automatically by the system; there are no restrictions here. When triggered by the user, they can enter corresponding commands in the dialogue input box on the client, select a control displayed on the client (e.g., control 91), or trigger it via voice commands, etc.

[0108] The above is a description of the intelligent question-answering system provided in the embodiments of this application. Based on the above content, an intelligent question-answering method provided in the embodiments of this application will be introduced below.

[0109] For example, Figure 10 This diagram illustrates a flowchart of an intelligent question-answering method provided in an embodiment of this application. It is understood that this method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. For example, the method can be executed by an intelligent question-answering device, which can be implemented by software and / or hardware, and can be configured in, but is not limited to, an electronic device or server; typically, it can be configured on a cloud computing platform. For ease of description, a cloud computing platform will be used as the execution subject in the following description. Figure 10 As shown, this intelligent question-answering method may include the following steps:

[0110] S1001. Perform a retrieval based on the first user query of the latest round of dialogue in the first session to obtain the first retrieval knowledge required in the latest round of dialogue.

[0111] In this embodiment, the client associated with the cloud computing platform can transmit all queries entered by the user in each round of dialogue within a session to the cloud computing platform. Upon receiving the query entered by the user in the latest round of dialogue, the cloud computing platform can perform a search in the knowledge base based on the user query in the latest round of dialogue to retrieve the first retrieval knowledge required for the latest round of dialogue. The knowledge base stores knowledge documents and / or knowledge fragments. In some embodiments, when the latest round of dialogue is the first round of dialogue in the current session, the cloud computing platform can directly perform a search in the knowledge base based on the user query and extract at least one sentence with the highest relevance to the user query from the search results to obtain the first retrieval knowledge. When the latest round of dialogue is a non-round dialogue in the current session, the cloud computing platform can first perform a search in the knowledge base based on the user query in the latest round of dialogue to obtain knowledge related to the user's query intent. Then, based on a second reference fragment, the cloud computing platform extracts at least one sentence with the highest relevance to the user query from the knowledge related to the user's query intent to filter out the first retrieval knowledge. The second referenced fragment is a segment of the retrieved knowledge needed in the previous round of dialogue that was referenced by the response text in the previous round of dialogue, and the quality of the second referenced fragment does not meet the requirements. Additionally, the first retrieved knowledge is weakly correlated with the second referenced fragment, or the first retrieved knowledge is strongly correlated with the second referenced fragment and has higher content richness than the second referenced fragment. For example, when the quality type of the second referenced fragment is incorrect, the first retrieved knowledge is weakly correlated with the second referenced fragment, thereby correcting the ineffective information gain of the retrieved knowledge; when the quality type of the second referenced fragment is insufficient content richness, the first retrieved knowledge is strongly correlated with the second referenced fragment and has higher content richness than the second referenced fragment, thereby optimizing the effective information gain of the retrieved knowledge. The process of obtaining the first retrieved knowledge can be detailed in the aforementioned descriptions of the retrieval module 324 and the dynamic chunking module 325, etc., and will not be repeated here.

[0112] S1002. The first retrieval knowledge and the first user query are processed through a neural network model to obtain the first response text.

[0113] In this embodiment, after obtaining the first retrieval knowledge, the first retrieval knowledge and the first user query can be processed by a neural network model to obtain the first response text. For example, the first retrieval knowledge and the first user query can be added to the Prompt module to obtain the Prompt; then, the Prompt is input into the neural network model for processing.

[0114] S1003. Based on the first retrieval knowledge, the first response text, and the user's first feedback on the first response text, perform a reflective evaluation on the first retrieval knowledge to obtain a first reflective evaluation conclusion. The first reflective evaluation conclusion is used to indicate the quality of the first cited fragment in the first retrieval knowledge. The first cited fragment is the fragment cited by the first response text.

[0115] In this embodiment, after receiving the first response text, the cloud computing platform can transmit the response text to its associated client so that the client can display the first response text to the user. After viewing the first response text, the user can provide corresponding feedback. For example, the user can ask follow-up questions in the next round of the current conversation, or the user query entered in the first round of the conversation in a new session, etc. Afterwards, the client can transmit the user's feedback on the first response text (i.e., the first feedback) to the cloud computing platform.

[0116] After receiving the first feedback, the cloud computing platform can perform a reflective evaluation of the first retrieved knowledge based on the first retrieved knowledge, the first response text, and the user's first feedback to the first response text, to obtain a first reflective evaluation conclusion. This first reflective evaluation conclusion can be used to indicate the quality of the first cited fragment in the first retrieved knowledge. The first cited fragment is the fragment cited by the first response text. In this way, the quality of the retrieved knowledge obtained from the knowledge base in the latest round of dialogue can be determined. In some embodiments, the cloud computing platform can first calculate the co-occurring fragments between the first retrieved knowledge and the first response text to obtain the first cited fragment. Then, the cloud computing platform can process the first cited fragment and the first feedback through a reflective evaluation model to obtain the first reflective evaluation conclusion. The process of obtaining the first reflective evaluation conclusion can be found in the aforementioned description of the reflective evaluation module 328, and will not be repeated here.

[0117] S1004. If the quality of the first quoted fragment does not meet the requirements, optimize the second retrieval knowledge required in the next round of dialogue based on the first quoted fragment. The second retrieval knowledge is obtained by retrieving the first quoted fragment and the second user query in the next round of dialogue.

[0118] In this embodiment, when the quality of the first reference fragment does not meet the requirements, the first retrieval knowledge cannot help the neural network model generate the correct response text. Therefore, the cloud computing platform can optimize the second retrieval knowledge required in the next round of dialogue based on the first reference fragment to help the neural network model generate the correct response text. The second retrieval knowledge is obtained by retrieving the first reference fragment and the second user query in the next round of dialogue. For details on the optimization process of the second retrieval knowledge, please refer to the aforementioned descriptions of the retrieval module 324 and the dynamic segmentation module 325, etc., which will not be repeated here.

[0119] S1005. If the quality of the first cited fragment meets the requirements, perform one or more of the following: store the first retrieval knowledge, generate a knowledge summary for the first session, or end the first session.

[0120] In this embodiment, when the quality of the first cited fragment does not meet the requirements, it indicates that the response text generated by the neural network model meets the user's needs. At this time, the cloud computing platform can perform one or more actions: storing the first retrieval knowledge, generating a knowledge summary for this session, or ending the session. Storing the first retrieval knowledge facilitates quickly finding accurate answers when encountering similar questions later, improving retrieval efficiency. Generating a knowledge summary for this session helps users understand the content of the session, enhancing the user experience.

[0121] In this way, during the intelligent question answering process, the quality of the retrieved knowledge generated in each round of dialogue is evaluated, and the poor-quality retrieved knowledge is corrected in the next round of dialogue. This can correct the erroneous knowledge in the retrieved knowledge in a timely manner, reduce the probability of introducing erroneous knowledge, and improve the answer quality of the neural network model.

[0122] It is understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, the various embodiments or technical features involved in the embodiments described above can be combined according to the actual situation, and the combined solution is still within the protection scope of this application.

[0123] Based on the methods in the above embodiments, this application also provides an intelligent question-answering device.

[0124] For example, Figure 11 A schematic diagram of the structure of an intelligent question-answering device provided in an embodiment of this application is shown. Figure 11As shown, the intelligent question-answering device 1100 includes a retrieval module 1101 and a processing module 1102. The retrieval module 1101 performs a retrieval based on the first user query of the latest round of dialogue in the first session to obtain the first retrieval knowledge required for the latest round of dialogue. The processing module 1102 processes the first retrieval knowledge and the first user query using a neural network model to obtain a first response text. The processing module 1102 is also used to perform a reflective evaluation of the first retrieval knowledge based on the first retrieval knowledge, the first response text, and the user's first feedback to the first response text, to obtain a first reflective evaluation conclusion. The first reflective evaluation conclusion indicates the quality of the first reference fragment in the first retrieval knowledge, where the first reference fragment is the fragment referenced by the first response text. The processing module 1102 is also used to optimize the second retrieval knowledge required for the next round of dialogue based on the first reference fragment if the quality of the first reference fragment does not meet the requirements. The second retrieval knowledge is obtained by retrieving the first reference fragment and the second user query in the next round of dialogue.

[0125] In some embodiments, the latest round of dialogue is a non-first round of dialogue in the current session. In this case, when the retrieval module 1101 performs a retrieval based on the first user query in the latest round of dialogue to obtain the first retrieval knowledge required for the latest round of dialogue, it specifically performs the following: based on the first user query, it performs a retrieval in a knowledge base to obtain knowledge related to the user's query intent, wherein the knowledge base stores knowledge documents and / or knowledge fragments; based on a second reference fragment, it filters the first retrieval knowledge from the knowledge related to the user's query intent, wherein the first retrieval knowledge is weakly correlated with the second reference fragment, or the first retrieval knowledge is strongly correlated with the second reference fragment and has a higher content richness than the second reference fragment. The second reference fragment is a fragment of the retrieval knowledge required in the previous round of dialogue that was referenced by the response text in the previous round of dialogue, and the quality of the second reference fragment does not meet the requirements.

[0126] In some embodiments, if the quality type of the second reference fragment is incorrect, the first retrieval knowledge is weakly correlated with the second reference fragment; if the quality type of the second reference fragment is lacking in content richness, the first retrieval knowledge is strongly correlated with the second reference fragment and its content richness is higher than that of the second reference fragment.

[0127] In some embodiments, the processing module 1102 is further configured to: if the quality of the first reference fragment meets the requirements, perform one or more of the following: store the first retrieval knowledge; generate a knowledge summary of the first session; or, terminate the first session.

[0128] In some embodiments, when the processing module 1102 performs a reflective evaluation on the first retrieval knowledge based on the first retrieval knowledge, the first response text, and the user's first feedback on the first response text to obtain a first reflective evaluation conclusion, it is specifically used to: calculate the co-occurrence fragments between the first retrieval knowledge and the first response text to obtain a first reference fragment; and process the first reference fragment and the first feedback through a reflective evaluation model to obtain the first reflective evaluation conclusion.

[0129] In some embodiments, the first feedback is a follow-up question from the user in the next round of conversation.

[0130] In some embodiments, Figure 11 The retrieval module 1101 and processing module 1102 shown can both be implemented in software or in hardware. For example, the implementation of retrieval module 1101 will be described below. Similarly, the implementation of processing module 1102 can refer to the implementation of retrieval module 1101.

[0131] As an example of a software functional unit, the retrieval module 1101 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the retrieval module 1101 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0132] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0133] As an example of a hardware functional unit, the retrieval module 1101 may include at least one computing device, such as a server. Alternatively, the retrieval module 1101 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The aforementioned PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0134] The multiple computing devices included in the retrieval module 1101 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the retrieval module 1101 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the retrieval module 1101 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0135] It should be noted that, in other embodiments, the retrieval module 1101 can be used to execute any step in the intelligent question-answering method described in the above embodiments, and the processing module 1102 can be used to execute any step in the intelligent question-answering method described in the above embodiments. Furthermore, the steps implemented by the retrieval module 1101 and the processing module 1102 can be specified as needed, and different steps in the intelligent question-answering method described in the above embodiments can be implemented by the retrieval module 1101 and the processing module 1102 respectively. Figure 11 The intelligent question-and-answer device 1100 shown has all the functions.

[0136] This application also provides a computing device 1200. For example... Figure 12 As shown, the computing device 1200 includes a bus 1202, a processor 1204, a memory 1206, and a communication interface 1208. The processor 1204, the memory 1206, and the communication interface 1208 communicate with each other via the bus 1202. The computing device 1200 can be a server or an electronic device. It should be understood that this application does not limit the number of processors and memories in the computing device 1200.

[0137] Bus 1202 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 12 The bus 1204 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1204 may include a path for transmitting information between various components of the computing device 1200 (e.g., memory 1206, processor 1204, communication interface 1208).

[0138] The processor 1204 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0139] The memory 1206 may include volatile memory, such as random access memory (RAM). The processor 1204 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0140] The memory 1206 stores executable program code, and the processor 1204 executes the executable program code to implement the aforementioned functions. Figure 11 The functions of the retrieval module 1101 and processing module 1102 shown are configured to implement the intelligent question-answering method described in the above embodiments. That is, the memory 1206 stores instructions for executing the intelligent question-answering method described in the above embodiments.

[0141] Alternatively, the memory 1206 stores executable code, and the processor 1204 executes the executable code to implement the aforementioned functions respectively. Figure 11 The intelligent question-answering device 1100 shown in the diagram performs the functions of the intelligent question-answering method described in the above embodiments. That is, the memory 1206 stores instructions for executing the intelligent question-answering method described in the above embodiments.

[0142] The communication interface 1203 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1200 and other devices or communication networks.

[0143] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. This computing device can be a server, such as a central server, an edge server, or a local server in a local data center.

[0144] like Figure 13 As shown, the computing device cluster includes at least one computing device 1200. The memory 1206 of one or more computing devices 1200 in the computing device cluster may store the same instructions for executing the intelligent question-answering method described in the above embodiments.

[0145] In some possible implementations, the memory 1206 of one or more computing devices 1200 in the computing device cluster may also store partial instructions for executing the intelligent question-answering method described in the above embodiments. In other words, a combination of one or more computing devices 1200 can jointly execute instructions for executing the intelligent question-answering method described in the above embodiments.

[0146] It should be noted that the memory 1206 in different computing devices 1200 within the computing device cluster can store different instructions, each used to execute the aforementioned instructions. Figure 11 The intelligent question-answering device 1100 shown contains some of its functions. That is, the instructions stored in the memory 1206 of different computing devices 1200 can realize the functions of one or more modules in the retrieval module 1101 and the processing module 1102.

[0147] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 14 One possible implementation is shown. For example... Figure 14 As shown, the two computing devices 1200A and 1200B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 1206 in computing device 1200A stores instructions for executing the functions of the retrieval module 1101. Simultaneously, the memory 1206 in computing device 1200B stores instructions for executing the functions of the processing module 1102.

[0148] It should be understood that Figure 14 The functions of the computing device 1200A shown can also be performed by multiple computing devices 1200. Similarly, the functions of the computing device 1200B can also be performed by multiple computing devices 1200.

[0149] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 13 and Figure 14 The connection method of the computing device cluster is different in that the memory 1206 of one or more computing devices 1200 in the computing device cluster can store the same instructions for executing the methods in the above embodiments.

[0150] In some possible implementations, the memory 1206 of one or more computing devices 1200 in the computing device cluster may also store partial instructions for executing the aforementioned methods. In other words, a combination of one or more computing devices 1200 can jointly execute the instructions for executing the aforementioned methods.

[0151] Based on the methods in the above embodiments, this application provides a computer-readable storage medium including computer program instructions. When the computer program instructions are executed by a computing device, the computing device performs the methods in the above embodiments; or, when the computer program instructions are executed by a cluster of computing devices, the cluster of computing devices performs the methods in the above embodiments. Exemplarily, the computer-readable storage medium can be any available medium that the computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0152] Based on the methods in the above embodiments, this application provides a computer program product containing instructions that, when executed by a computing device, cause the computing device to perform the methods in the above embodiments, or, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the methods in the above embodiments.

[0153] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0154] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0155] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, cloud computing platform, or data center to another website, computer, cloud computing platform, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a cloud computing platform or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0156] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0157] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. An intelligent question-answering method, characterized in that, The method includes: A retrieval is performed based on the first user query in the latest round of dialogue in the first session to obtain the first retrieval knowledge required in the latest round of dialogue; The first retrieval knowledge and the first user query are processed by a neural network model to obtain the first response text; Based on the first retrieved knowledge, the first response text, and the user's first feedback on the first response text, the first retrieved knowledge is reflected upon and evaluated to obtain a first reflection evaluation conclusion. The first reflection evaluation conclusion is used to indicate the quality of the first cited fragment in the first retrieved knowledge. The first cited fragment is the fragment cited by the first response text. If the quality of the first reference fragment does not meet the requirements, the second retrieval knowledge required in the next round of dialogue is optimized based on the first reference fragment, wherein the second retrieval knowledge is obtained by retrieving the first reference fragment and the second user query in the next round of dialogue.

2. The method according to claim 1, characterized in that, The latest round of dialogue is a non-first round of dialogue in the first session; The retrieval based on the first user query in the latest round of dialogue to obtain the first retrieval knowledge required in the latest round of dialogue includes: Based on the first user query, a search is performed in the knowledge base to obtain knowledge related to the user's query intent, wherein the knowledge base stores knowledge documents and / or knowledge fragments; Based on the second reference fragment, the first retrieval knowledge is filtered from the knowledge related to the user's query intent. The first retrieval knowledge is weakly related to the second reference fragment, or the first retrieval knowledge is strongly related to the second reference fragment and has a higher content richness than the second reference fragment. The second reference fragment is a fragment of the retrieval knowledge required in the previous round of dialogue that is referenced by the response text in the previous round of dialogue, and the quality of the second reference fragment does not meet the requirements.

3. The method according to claim 2, characterized in that, If the quality type of the second reference fragment is incorrect, the first retrieval knowledge is weakly correlated with the second reference fragment. When the quality type of the second cited fragment is lacking in content richness, the first retrieved knowledge is strongly correlated with the second cited fragment and has a higher content richness than the second cited fragment.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: If the quality of the first referenced fragment meets the requirements, perform one or more of the following: Store the first retrieved knowledge; Generate a knowledge summary of the first session; Alternatively, end the first session.

5. The method according to any one of claims 1-4, characterized in that, The step of reflecting on and evaluating the first retrieval knowledge based on the first retrieval knowledge, the first response text, and the user's first feedback on the first response text to obtain a first reflection and evaluation conclusion includes: Calculate the co-occurrence segments between the first retrieved knowledge and the first response text to obtain the first cited segment; The first quoted fragment and the first feedback are processed by the reflection and evaluation model to obtain the first reflection and evaluation conclusion.

6. The method according to any one of claims 1-5, characterized in that, The first feedback is a follow-up question from the user in the next round of conversation.

7. An intelligent question-and-answer device, characterized in that, include: The retrieval module performs a retrieval based on the first user query in the latest round of dialogue in the first session, so as to obtain the first retrieval knowledge required in the latest round of dialogue; The processing module is used to process the first retrieved knowledge and the first user query through a neural network model to obtain the first response text; The processing module is further configured to reflect on and evaluate the first retrieval knowledge based on the first retrieval knowledge, the first response text, and the user's first feedback to the first response text, so as to obtain a first reflection and evaluation conclusion. The first reflection and evaluation conclusion is used to indicate the quality of the first reference fragment in the first retrieval knowledge. The first reference fragment is the fragment referenced by the first response text. The processing module is further configured to optimize the second retrieval knowledge required in the next round of dialogue based on the first retrieval fragment if the quality of the first retrieval fragment does not meet the requirements. The second retrieval knowledge is obtained by retrieving the first retrieval fragment and the second user query in the next round of dialogue.

8. The apparatus according to claim 7, characterized in that, The latest round of dialogue is a non-first round of dialogue in this session; When the retrieval module performs a retrieval based on the first user query in the latest round of dialogue to obtain the first retrieval knowledge required in the latest round of dialogue, it is specifically used for: Based on the first user query, a search is performed in the knowledge base to obtain knowledge related to the user's query intent, wherein the knowledge base stores knowledge documents and / or knowledge fragments; Based on the second reference fragment, the first retrieval knowledge is filtered from the knowledge related to the user's query intent. The first retrieval knowledge is weakly related to the second reference fragment, or the first retrieval knowledge is strongly related to the second reference fragment and has a higher content richness than the second reference fragment. The second reference fragment is a fragment of the retrieval knowledge required in the previous round of dialogue that is referenced by the response text in the previous round of dialogue, and the quality of the second reference fragment does not meet the requirements.

9. The apparatus according to claim 8, characterized in that, If the quality type of the second reference fragment is incorrect, the first retrieval knowledge is weakly correlated with the second reference fragment. When the quality type of the second cited fragment is lacking in content richness, the first retrieved knowledge is strongly correlated with the second cited fragment and has a higher content richness than the second cited fragment.

10. The apparatus according to any one of claims 7-9, characterized in that, The processing module is further configured to: If the quality of the first referenced fragment meets the requirements, perform one or more of the following: Store the first retrieved knowledge; Generate a knowledge summary of the first session; Alternatively, end the first session.

11. The apparatus according to any one of claims 7-10, characterized in that, When the processing module performs a reflective evaluation on the first retrieval knowledge based on the first retrieval knowledge, the first response text, and the user's first feedback on the first response text to obtain a first reflective evaluation conclusion, it is specifically used for: Calculate the co-occurrence segments between the first retrieved knowledge and the first response text to obtain the first cited segment; The first quoted fragment and the first feedback are processed by the reflection and evaluation model to obtain the first reflection and evaluation conclusion.

12. The apparatus according to any one of claims 7-11, characterized in that, The first feedback is a follow-up question from the user in the next round of conversation.

13. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-6.

14. A computer-readable storage medium, characterized in that, The method includes computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any one of claims 1-6, wherein the cluster of computing devices includes at least one computing device.

15. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1-6, wherein the computing device cluster includes at least one computing device.

Images