Intelligent question and answering method and apparatus, and computing device cluster

Abstract

An intelligent question and answering method, comprising: performing retrieval on the basis of a first user query in a current conversational turn, so as to obtain first retrieved knowledge required in the current conversational turn; processing the first retrieved knowledge and the first user query to obtain response text; on the basis of the first retrieved knowledge, the response text, and user feedback to the response text, performing a reflective evaluation on the first retrieved knowledge, so as to obtain a reflective evaluation conclusion, the reflective evaluation conclusion being used for indicating quality of a cited snippet in the first retrieved knowledge, and the cited snippet being a snippet referenced by the response text; when the quality of the cited snippet does not satisfy a requirement, optimizing, on the basis of the cited snippet, second retrieved knowledge required in a next conversational turn, the second retrieved knowledge being obtained by performing retrieval on the basis of a first cited snippet and a second user query in the next conversational turn. The method allows for erroneous knowledge in retrieved knowledge to be corrected in a timely manner, thereby reducing the probability of introducing erroneous knowledge, and improving answer quality of neural network models.

Description

Intelligent question and answer method, device and computing device cluster

[0001] The present application claims priority to the Chinese patent application No. 202411362670.8, filed on September 27, 2024, entitled "Intelligent question and answer method, device and computing device cluster", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to an intelligent question and answer method, device and computing device cluster. BACKGROUND

[0003] With the rapid development of large language models (LLMs), a large number of chat robots have appeared in the industry, such as chat generative pre-trained transformers (ChatGPT) and the like. These chat robots take LLMs as the foundation and can complete complex tasks such as question and answer, copywriting, and code generation through dialogue with users. Although LLMs show strong language understanding and summarization capabilities, the training data of LLMs is usually general Internet data, lacking the latest knowledge and private domain data in the vertical field, so LLMs often have hallucinations when facing specific domain or professional problems.

[0004] In order to improve the accuracy, timeliness and professionalism of LLMs, the current mainstream approach is retrieval augmented generation (RAG) technology. RAG first constructs a knowledge document library, when a user raises a query question, retrieves the most relevant knowledge text from the knowledge document library, and inputs the user query and the retrieved knowledge text into the LLM, and returns the response answer generated by the LLM to the user. In this process, RAG provides additional knowledge information for LLM to better understand the query and create a response. RAG can introduce external professional knowledge for LLM by providing relevant knowledge text to assist LLM to generate higher quality answers. However, if RAG provides incorrect external knowledge or irrelevant external information when answering questions, it not only cannot improve the quality of LLM answers, but also may exacerbate the hallucination effect of LLM. Therefore, how to timely correct and optimize the quality of the retrieved knowledge in the multi-round dialogue process, reduce the introduction of incorrect knowledge, and ensure that LLM can generate accurate answers is a technical problem that needs to be solved at present. SUMMARY

[0005] The application provides an intelligent question answering method and device, a computing device cluster, a computer storage medium and a computer product, which can correct and optimize the retrieval knowledge with poor quality in a multi-round dialogue process, reduce the introduction of incorrect knowledge, and ensure that the LLM can generate accurate answers.

[0006] In a first aspect, the application provides an intelligent question answering method, which includes: performing retrieval based on a first user query in a latest round of dialogue in a first session to obtain first retrieval knowledge required in the latest round of dialogue; processing the first retrieval knowledge and the first user query through a neural network model to obtain a first response text; performing reflection evaluation on the first retrieval knowledge based on the first retrieval knowledge, the first response text and first feedback of a user on the first response text to obtain a first reflection evaluation conclusion, the first reflection evaluation conclusion being used to indicate the quality of a first cited segment in the first retrieval knowledge, the first cited segment being a segment cited by the first response text; and in the case that the quality of the first cited segment does not meet the requirements, performing optimization on second retrieval knowledge required in a next round of dialogue based on the first cited segment, wherein the second retrieval knowledge is obtained based on the first cited segment and a second user query in the next round of dialogue.

[0007] In this way, the quality of the retrieval knowledge generated for each round of dialogue is evaluated in the intelligent question answering process, and the retrieval knowledge with poor quality is corrected in the next round of dialogue, so that incorrect knowledge in the retrieval knowledge can be corrected in time, the probability of introducing incorrect knowledge is reduced, and the answer quality of the neural network model is improved.

[0008] In a possible implementation, the latest round of dialogue is a non-first round of dialogue in the first session. At this time, performing retrieval based on the first user query in the latest round of dialogue to obtain the first retrieval knowledge required in the latest round of dialogue includes: performing retrieval in a knowledge base based on the first user query to obtain knowledge related to the user query intention, wherein the knowledge base stores knowledge documents and / or knowledge segments; and screening the first retrieval knowledge from the knowledge related to the user query intention based on a second cited segment, the first retrieval knowledge being weakly related to the second cited segment, or the first retrieval knowledge being strongly related to the second cited segment and having a content richness higher than that of the second cited segment. The second cited segment is a segment in retrieval knowledge required in a previous round of dialogue that is cited by a response text in the previous round of dialogue, and the quality of the second cited segment does not meet the requirements. In this way, incorrect knowledge can be excluded in the process of generating retrieval knowledge in the current round, and the retrieval knowledge can be corrected in time; and the retrieval knowledge can be enriched in time in the process of generating retrieval knowledge in the current round, and the retrieval knowledge can be optimized in time.

[0009] In a possible implementation, the first search knowledge is weakly related to the second cited segment when the quality type of the second cited segment is incorrect. In this way, the purpose of correcting the invalid information gain of the search knowledge can be achieved. When the quality type of the second cited segment is lack of content richness, the first search knowledge is strongly related to the second cited segment and has higher content richness than the second cited segment. In this way, the purpose of optimizing the effective information gain of the search knowledge can be achieved.

[0010] In a possible implementation, the method further includes: when the quality of the first cited segment meets the requirement, performing one or more of the following: storing the first search knowledge; generating a knowledge summary of the first session; or ending the first session. After the first search knowledge is stored, accurate answers can be quickly found when similar problems are encountered subsequently, and the search efficiency is improved. Generating the knowledge summary of the current session can facilitate the user to understand the content of the current session and improve the user interaction experience. In addition, after the first session is ended, a new round of session can be started.

[0011] In a possible implementation, based on the first search knowledge, the first response text, and the first feedback of the user to the first response text, the first search knowledge is rethought and evaluated to obtain a first rethought evaluation conclusion, including: calculating a co-occurrence segment between the first search knowledge and the first response text to obtain a first cited segment; and processing the first cited segment and the first feedback by using a rethought evaluation model to obtain the first rethought evaluation conclusion. Since the response text is obtained based on the search knowledge, and some segments in the search knowledge are cited in the response text, the quality of the search knowledge can be accurately evaluated by evaluating the cited segments in the response text.

[0012] In a possible implementation, the first feedback is a follow-up question of the user in the next round of dialogue.

[0013] In a second aspect, the present application provides an intelligent question-answering device, comprising: a retrieval module and a processing module. The retrieval module is configured to perform retrieval based on a first user query in a latest round of dialogue in a first session to obtain first retrieval knowledge required in the latest round of dialogue. The processing module is configured to perform processing on the first retrieval knowledge and the first user query by using a neural network model to obtain a first response text. The processing module is further configured to perform reflection evaluation on the first retrieval knowledge based on the first retrieval knowledge, the first response text, and first feedback of the user on the first response text to obtain a first reflection evaluation conclusion, the first reflection evaluation conclusion being used to indicate a quality of a first cited segment in the first retrieval knowledge, the first cited segment being a segment cited by the first response text. The processing module is further configured to perform optimization on second retrieval knowledge required in a next round of dialogue based on the first cited segment in a case where the quality of the first cited segment does not meet a requirement, the second retrieval knowledge being obtained based on the first cited segment and a second user query in the next round of dialogue.

[0014] In a possible implementation, the latest round of dialogue is a non-first round of dialogue in the current session. In this case, when performing retrieval based on the first user query in the latest round of dialogue to obtain the first retrieval knowledge required in the latest round of dialogue, the retrieval module is specifically configured to: perform retrieval in a knowledge base based on the first user query to obtain knowledge related to an intention of the user query, wherein the knowledge base stores knowledge documents and / or knowledge segments; and select the first retrieval knowledge from the knowledge related to the intention of the user query based on a second cited segment, the first retrieval knowledge being weakly related to the second cited segment, or the first retrieval knowledge being strongly related to the second cited segment and having a content richness higher than that of the second cited segment. The second cited segment is a segment in retrieval knowledge required in a previous round of dialogue that is cited by a response text in the previous round of dialogue, and the quality of the second cited segment does not meet the requirement.

[0015] In a possible implementation, in a case where the quality type of the second cited segment is incorrect, the first retrieval knowledge is weakly related to the second cited segment; and in a case where the quality type of the second cited segment is a lack of content richness, the first retrieval knowledge is strongly related to the second cited segment and has a content richness higher than that of the second cited segment.

[0016] In a possible implementation, the processing module is further configured to: in a case where the quality of the first cited segment meets the requirement, perform one or more of the following: store the first retrieval knowledge; generate a knowledge summary of the first session; or end the first session.

[0017] In a possible implementation, when the processing module performs the reflection evaluation on the first retrieval knowledge based on the first retrieval knowledge, the first response text, and the first feedback of the user on the first response text to obtain a first reflection evaluation conclusion, the processing module is specifically configured to: calculate a co-occurrence segment between the first retrieval knowledge and the first response text to obtain a first reference segment; and perform processing on the first reference segment and the first feedback by using a reflection evaluation model to obtain the first reflection evaluation conclusion.

[0018] In a possible implementation, the first feedback is a follow-up question of the user in the next round of dialogue.

[0019] In a third aspect, the present application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method described in the first aspect or any possible implementation of the first aspect.

[0020] In a fourth aspect, the present application provides a computer-readable storage medium, including computer program instructions, when the computer program instructions are executed by a computing device, the computing device executes the method described in the first aspect or any possible implementation of the first aspect; or, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method described in the first aspect or any possible implementation of the first aspect. Illustratively, the computing device cluster can include one or more computing devices.

[0021] In a fifth aspect, the present application provides a computer program product including instructions, when the instructions are executed by a computing device, the computing device executes the method described in the first aspect or any possible implementation of the first aspect, or, when the instructions are executed by a computing device cluster, the computing device cluster executes the method described in the first aspect or any possible implementation of the first aspect. Illustratively, the computing device cluster can include one or more computing devices.

[0022] It can be understood that the beneficial effects of the above-mentioned second aspect to the fifth aspect can be referred to the related description in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0023] FIG. 1 is an architecture schematic diagram of an intelligent question answering system according to an embodiment of the present application;

[0024] FIG. 2 is a technical concept schematic diagram of an intelligent question answering according to an embodiment of the present application;

[0025] FIG. 3 is an architecture schematic diagram of an intelligent question answering system according to an embodiment of the present application;

[0026] FIG. 4 is a schematic diagram of a process for obtaining a reflection evaluation conclusion according to an embodiment of the present application;

[0027] FIG. 5 is a schematic diagram of software and hardware of an intelligent question-answering system according to an embodiment of the present application;

[0028] FIG. 6 is a schematic diagram of a work flow of an intelligent question-answering system according to an embodiment of the present application;

[0029] FIG. 7 is a schematic diagram of product implementation of software modules of an intelligent question-answering system according to an embodiment of the present application;

[0030] FIG. 8 is a schematic diagram of flow and operation of data in an intelligent question-answering system according to an embodiment of the present application;

[0031] FIG. 9 is a schematic diagram of an interface provided by a client in a conversation according to an embodiment of the present application;

[0032] FIG. 10 is a schematic diagram of a flow of an intelligent question-answering method according to an embodiment of the present application;

[0033] FIG. 11 is a schematic diagram of a structure of an intelligent question-answering apparatus according to an embodiment of the present application;

[0034] FIG. 12 is a schematic diagram of a structure of a computing device according to an embodiment of the present application;

[0035] FIG. 13 is a schematic diagram of a structure of a computing device cluster according to an embodiment of the present application;

[0036] FIG. 14 is a schematic diagram of a structure of another computing device cluster according to an embodiment of the present application. DETAILED DESCRIPTION

[0037] The term “and / or” herein is used to describe an association relationship between associated objects, which means that there can be three relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone. The symbol “ / ” herein represents an or relationship of associated objects, for example, A / B represents A or B.

[0038] The terms “first” and “second” and the like in the specification and claims herein are used to distinguish different objects, and are not used to describe a specific order of the objects. For example, the first response message and the second response message are used to distinguish different response messages, and are not used to describe a specific order of the response messages.

[0039] In the embodiments of the present application, the word "exemplary" or "for example" is used to mean serving as an example, instance, or illustration, and not necessarily to imply any preference or superiority. In fact, an "exemplary" or "for example" embodiment should not necessarily be considered to have any advantage over other embodiments.

[0040] In the description of the embodiments of the present application, unless otherwise specified, "a plurality of" means two or more, for example, a plurality of processing units means two or more processing units, and the like; a plurality of elements means two or more elements, and the like.

[0041] Firstly, the related technical terms involved in the technical solutions provided in the present application are introduced.

[0042] (1) Retrieval augmented generation (RAG)

[0043] RAG is a technology that combines retrieval and generation to improve the performance of LLM in information retrieval and text generation tasks. RAG enhances the ability of the generation model by introducing a retrieval mechanism to provide external retrieval content before the large model generates an answer, thereby improving the accuracy and relevance of the generated text.

[0044] (2) Hallucination

[0045] Hallucination refers to false or incorrect information generated by LLM when generating text. This information does not conform to the facts or is inconsistent with the knowledge of the real world.

[0046] (3) Dynamic chunking

[0047] In the RAG technology, chunking is a key step for dividing large texts or documents into smaller and more manageable chunks. A document can be very large, and dividing it into smaller chunks can improve retrieval efficiency and generation quality to meet the length limit requirements of large models. In related technologies, offline chunking is usually used: that is, the documents in the prepared knowledge base are chunked in the offline link. This approach generally divides the chunks according to the length of the text, and the content of the chunk does not change after the document is divided into chunks. In the present application, dynamic chunking technology is used, that is, the documents are not divided into smaller chunks in the offline link, but the results of chunking are dynamically determined according to the dialogue scenario in the online process.

[0048] (4) Reflection evaluation

[0049] Reflection evaluation generally refers to evaluating the result generated by the LLM to determine whether the generated answer meets the user's query. In this application, it mainly refers to evaluating the retrieved knowledge through the RAG technology to evaluate whether the retrieved knowledge helps the generated content of the LLM and whether it meets the user's query requirements.

[0050] The technical solutions provided in the present application are described below.

[0051] For example, FIG. 1 shows an architecture diagram of an intelligent question answering system provided by an embodiment of the present application. As shown in FIG. 1, the intelligent question answering system can include a knowledge base construction module 110, a query understanding module 120, a retrieval module 130, a prompt (Prompt) generation module 140, an LLM 150, and a response evaluation module 160. The knowledge base construction module 110 is mainly used to divide the prepared knowledge documents into blocks in the offline link to build a knowledge base in advance in the offline link for use by the retrieval module 130. The knowledge base stores knowledge segments divided from the knowledge documents. The query understanding module 120 is mainly used to understand the historical question and answer and the user query in the current round of dialogue, including rewriting, expansion, etc., to rewrite the original user query into an optimized query text. The retrieval module 130 is mainly used to recall the most relevant knowledge segments from the knowledge base to the user query intent to obtain corresponding retrieval knowledge. The Prompt generation module 140 is mainly used to assemble the retrieval knowledge and the query text output by the query understanding module 120 into a Prompt and input it to the LLM 150. The LLM 150 is mainly used to make inferences based on the Prompt from the Prompt generation module 140 to generate a response. The response evaluation module 160 is mainly used to evaluate the response result of the LLM 150 to determine whether it meets the user's query requirements, optimize the response result, and obtain the final response content.

[0052] The intelligent question answering system shown in FIG. 1, although it can effectively realize the integration and retrieval function of the external knowledge base and solve the problem of the lack of professional knowledge in the answer of the LLM, however, when evaluating the quality of the answer content generated by the LLM, it can only evaluate the answer effect of the final LLM and cannot directly reflect the quality of the RAG retrieved external knowledge. Therefore, in the process of using the intelligent question system shown in FIG. 1 for multi-round dialogue, it is difficult to correct the retrieval knowledge with poor quality in time, which makes it difficult to avoid introducing incorrect knowledge.

[0053] Therefore, the present application provides an intelligent question answering method, which can correct and optimize the retrieval knowledge with poor quality in time during the process of using the LLM to answer questions, reduce the probability of introducing incorrect knowledge, and improve the answer quality of the LLM.

[0054] Exemplarily, FIG. 2 shows a schematic diagram of a technical concept of the intelligent question-answering provided by the embodiments of the present application. Under the technical concept, the quality of the generated retrieval knowledge in the current round can be directly evaluated, and when the quality of the generated retrieval knowledge in the current round does not meet the requirements, the incorrect knowledge in the retrieval knowledge can be corrected in the next round of dialogue. As shown in FIG. 2, under the technical concept of the intelligent question-answering, four parts are mainly included, which are respectively: retrieval, dynamic blocking, generating response and reflection evaluation.

[0055] In the retrieval part, knowledge documents and / or knowledge segments consistent with the user input at each round of dialogue in a session are retrieved from the knowledge base based on the user input at each round of dialogue. In the dynamic chunking part, sentences meeting the requirements are extracted from the retrieved knowledge documents and / or knowledge segments, and the extracted sentences are grouped into text chunks as retrieval knowledge during each round of dialogue. In the response generation part, the user input in each round of dialogue and the generated retrieval knowledge are assembled into a Prompt, which is transmitted to the LLM to generate a response (i.e., obtain LLM generated content). In the reflection evaluation part, the quality of the retrieval knowledge and the LLM generated content is reflected and evaluated based on the generated retrieval knowledge in each round of dialogue and the user feedback (i.e., user feedback) on the LLM generated content during the corresponding round of dialogue, to obtain a reflection evaluation conclusion. The reflection evaluation conclusion can be used to indicate the quality of the segment in the retrieval knowledge cited by the LLM generated content. In this embodiment, a session is composed of at least one round of dialogue, and each round of dialogue can be a complete question and answer session. For example, the user asks, “What is the weather tomorrow?” and the system answers, “Tomorrow is expected to be sunny with a high of 25 degrees Celsius.” The user further asks, “What is the temperature?” and the system supplements, “The maximum temperature is about 25 degrees Celsius.” This can be understood as a session, and the user inquiry and the corresponding answer of the system can be understood as a round of dialogue. In addition, multiple sessions can be included in a session window, and different sessions involve different topics. For example, in a session window, the user can first have a dialogue related to the weather and then have a dialogue related to shopping. At this time, the dialogue related to the weather can be considered as a session, and the dialogue related to shopping can be considered as another session. In addition, when the reflection evaluation conclusion generated in a round of dialogue indicates that the quality of the segment in the retrieval knowledge cited by the LLM generated content meets the requirements, it can also be considered that a session is completed. For example, a session window includes 15 rounds of dialogue, and the reflection evaluation conclusions generated in the first to ninth rounds of dialogue all indicate that the quality of the segment in the retrieval knowledge cited by the LLM generated content does not meet the requirements, while the reflection evaluation conclusion generated in the tenth round of dialogue indicates that the quality of the segment in the retrieval knowledge cited by the LLM generated content meets the requirements. It is considered that the first to tenth rounds of dialogue are a session, and the eleventh round of dialogue belongs to a new session, and the eleventh round of dialogue is the first round of dialogue in the latest session. At this time, the topic involved in the first to tenth rounds of dialogue can be the same as the topic involved in the eleventh round of dialogue, or it can be different. In addition, the sessions involved in different session windows belong to different sessions.

[0056] Under the technical concept shown in FIG. 2, when the quality of the segment of the retrieved knowledge indicated by the reflection evaluation conclusion obtained in the current round of dialogue meets the requirements, it indicates that the correct retrieved knowledge and LLM generated content have been obtained, and thus the current session can be ended. When the quality of the segment of the retrieved knowledge indicated by the reflection evaluation conclusion obtained in the current round of dialogue does not meet the requirements, it indicates that there is a segment of the retrieved knowledge whose quality does not meet the requirements, which will introduce incorrect knowledge. In order to correct the incorrect knowledge, the next round of dialogue can be performed. In the dynamic partitioning part of the current round, in the process of correcting the segment of the retrieved knowledge whose quality does not meet the requirements generated in the previous round, the sentences weakly related to the segment of the retrieved knowledge whose quality does not meet the requirements generated in the previous round can be extracted from the knowledge documents and / or knowledge segments retrieved in the current round as the retrieved knowledge, so as to correct the invalid information gain of the retrieved knowledge. In this way, in the dynamic partitioning process, the quality of the retrieved knowledge can be corrected in time and the retrieved knowledge with high quality can be generated through the guidance of the reflection evaluation conclusion of the previous round.

[0057] With reference to FIG. 2, it is assumed that a session includes two rounds of dialogue. In the first round of dialogue, when performing retrieval, the query text obtained by rewriting or expanding the original query of the user through query understanding can be used to perform retrieval from the knowledge base to retrieve N knowledge documents (and / or knowledge segments) with the highest relevance scores with the query text. Since there is no reflection evaluation conclusion to be referred to in the first round of dialogue, the semantic relevance algorithm can be used to calculate the semantic relevance scores of each sentence in the N knowledge documents (and / or knowledge segments) and the query text, and the Top K sentences with the highest scores can be selected as the text content meeting the requirements and combined to form the final retrieved knowledge. Then, the retrieved knowledge and the query text can be assembled into a Prompt and input to the LLM to generate a response text, so as to obtain the LLM generated content. Finally, the quality of the retrieved knowledge and the LLM generated content can be reflected and evaluated according to the retrieved knowledge in the current round, the LLM generated content (i.e., the generated response), and the feedback of the user to the LLM generated content in the current round (i.e., the user feedback). In the process of reflection evaluation, the source of the reference can be detected to identify the segment of the retrieved knowledge referenced by the LLM generated content (hereinafter referred to as “reference segment”); then, the quality of the reference segment can be reflected and evaluated to determine whether the referenced retrieved knowledge meets the query requirements of the user, and the reflection evaluation conclusion is recorded as the optimization basis for the next round of dynamic partitioning. It is assumed that the reflection evaluation conclusion in the current round indicates that the quality of the segment of the retrieved knowledge referenced by the LLM generated content does not meet the requirements, and thus the next round of dialogue can be entered.

[0058] In the second round of dialogue, when doing retrieval, the query text obtained by rewriting or expanding the original query of the user by query understanding can be used to do retrieval from the knowledge base to retrieve the N knowledge documents (and / or knowledge fragments) with the highest relevance score with the query text. Since there is a reference evaluation conclusion of the previous round of dialogue, the text content weakly related to the poor quality reference fragment indicated by the reflection evaluation conclusion of the previous round can be extracted from the N knowledge documents (and / or knowledge fragments) in combination with the reflection evaluation conclusion of the previous round, and the final retrieval knowledge is composed, so as to correct the invalid information gain of the retrieval knowledge.

[0059] After obtaining the retrieval knowledge in the second round of dialogue, the retrieval knowledge and the query text can be assembled into Prompt, input into the LLM to generate response text, so as to obtain the LLM generated content. Finally, the quality of the retrieval knowledge and the LLM generated content can be reflected and evaluated according to the retrieval knowledge retrieved in the current round, the LLM generated content and the user feedback (i.e. user feedback) on the LLM generated content in the current round. Assuming that the reflection evaluation conclusion of the current round indicates that the quality of the fragment cited by the LLM generated content in the retrieval knowledge meets the requirements, the current session can be ended at this time.

[0060] Under the technical concept of intelligent question answering shown in FIG. 2, the retrieval knowledge in the process of one round of dialogue is extracted from the retrieved knowledge documents (and / or knowledge fragments), and in the extraction process, the reflection evaluation conclusion of the retrieval knowledge in the previous round can be used for guidance, so as to know where the retrieval knowledge is wrong, and then as much as possible to consider the data different from the retrieval knowledge content in the previous round in the current round, to realize timely correction of the retrieval knowledge with poor quality, improve the quality of the retrieval knowledge extracted in the current round, greatly improve the answering ability of the LLM, and reduce the number of rounds required for user question and answer. In addition, since the quality of the retrieval knowledge and the response result of the LLM are considered simultaneously in the reflection evaluation process, and the user feedback is used as the evaluation basis, rather than only using the ability of the LLM itself to make a judgment, the reflection evaluation conclusion can directly reflect the quality of the RAG external knowledge, so that the reflection evaluation conclusion of the retrieval knowledge in the previous round can be used as the optimization basis for dynamic block in the current round.

[0061] The specific implementation process of the above technical concept is described below.

[0062] For example, FIG. 3 shows an architecture schematic diagram of an intelligent question answering system provided by an embodiment of the present application. As shown in FIG. 3, the intelligent question answering system 300 can include a client 310 and a server 320.

[0063] The client 310 can be a desktop application, a mobile application, a web application, or a web-based application, etc., which can be deployed on an electronic device such as a mobile phone, a computer, etc. The client 310 can provide a user interface (UI) for a user to interact with the client 310, such as inputting content of a query desired by the user on the client 310, or viewing content generated by the intelligent question-answering system 300, etc. The user interface can be a graphical user interface (GUI), a command-line interface (CLI), or other forms of interactive interface, which is not limited herein. In the embodiments, the client 310 can transmit a query of a user (i.e., a user query) to the server 320, and show the user the LLM generated content returned by the server 320. In some embodiments, the client 310 can also show guidance information when showing the LLM generated content returned by the server 320. The guidance information is used to guide the user to provide evaluation feedback on the retrieved knowledge and / or the LLM generated content of the current round. The user can optionally use the guidance information to generate query text for the next round of dialogue, or can use other queries to ask follow-up questions in the next round, which is not limited herein.

[0064] The server 320 can be, but is not limited to, a server or a cloud server, or other devices or platforms that can provide at least computing resources. The server 320 can be configured with, but is not limited to, a knowledge base construction module 321, a knowledge base 322, a query understanding module 323, a retrieval module 324, a dynamic chunking module 325, a Prompt generation module 326, an LLM 327, and a reflection evaluation module 328.

[0065] The knowledge base construction module 321 is mainly used to build the knowledge base 322 in advance in the offline link for the retrieval module 324. In the knowledge base construction process, the knowledge base construction module 321 can retain the original content of each knowledge document and does not perform the blocking operation in advance, but directly stores each knowledge document into the inverted index or vector index library and the like, that is, stores the original knowledge document into the inverted index or vector index library and the like in the document granularity. For example, the knowledge document is in the format of PDF, PPTX, Docx, txt, HTML, Markdown and the like. Of course, the knowledge base construction module 321 can also perform the blocking on the prepared knowledge document in the offline link, and store each text block into the inverted index or vector index library and the like, or store a part of the knowledge document into the knowledge base in the document granularity, and store another part of the knowledge document into the knowledge base after blocking, which can be determined according to the actual situation, and is not limited here. In some embodiments, considering that one text segment after blocking usually only has partial content of one knowledge document, the article content integrity and the theme consistency are lost, which increases the difficulty of retrieving the correct knowledge, therefore, the knowledge document can be stored in the document granularity in priority, so as to retain the complete content of the knowledge document, avoid losing the article content integrity and the theme consistency, and reduce the difficulty of retrieving the correct knowledge.

[0066] The knowledge base 322 is mainly used to store the knowledge document in the document and / or text block granularity by the inverted index or vector index and the like.

[0067] The query understanding module 323 is mainly used to rewrite or expand the original query (i.e., the query from the client 310) of the user in the current round (i.e., the current round) and the like, and rewrite the original query of the user into the optimized query text, so as to make the content more fluent. For example, when the user query is “How is the weather”, when the user allows the system to obtain the current location, the query understanding module 323 can optimize the user query into “How is the weather in Beijing today”. For example, the query understanding module 323 can combine the historical query and the historical answer to optimize the original query in the current round, so as to supplement the necessary information mentioned in the context, eliminate the ambiguity, and make the content more fluent.

[0068] The retrieval module 324 is mainly used to recall N knowledge documents (and / or knowledge segments) with the highest relevance score from the knowledge base 322 according to the user query. For the convenience of description, the “knowledge document and / or knowledge segment” is collectively referred to as “knowledge data” below. For example, the retrieval module 324 can but not limited to calculate the similarity between the optimized query text and the knowledge data stored in the knowledge base 322 by the cosine similarity algorithm and the like, and then recall N knowledge data with the highest similarity as the data in this time.

[0069] The dynamic chunking module 325 is mainly used for dynamically chunking the N pieces of knowledge data retrieved by the retrieval module 324 to form the final retrieval knowledge. In the first round of dialogue process of a session, the dynamic chunking module 325 can use semantic relevance algorithm to calculate the semantic relevance scores of each sentence in the query text and the N pieces of knowledge data, and select the Top K sentences with the highest scores as the qualified text content, and form a text chunk as the retrieval knowledge. In the non-first round of dialogue process of a session, the dynamic chunking module 325 can use semantic relevance algorithm to calculate the semantic relevance scores of each sentence in the query text and the N pieces of knowledge data, and select the Top M sentences with the highest scores. Then, the dynamic chunking module 325 combines the reflection evaluation conclusion of the retrieval knowledge generated in the last round of dialogue provided by the reflection evaluation module 328 to filter out the Top K (of course, other quantities are also possible, which are not limited here) sentences from the Top M sentences, and take them as the retrieval knowledge, so as to correct the retrieval knowledge with poor quality in time. In some embodiments, when the quality of the fragment cited by the LLM generated content in the retrieval knowledge does not meet the requirements, the corresponding cited fragment is not necessarily completely wrong, but may be not detailed, etc. Therefore, if the retrieval knowledge is extracted from the candidate sentences, all the fragments with quality not meeting the requirements in the last round of retrieval knowledge are removed, which may cause overcorrection, and thus the quality of the retrieval knowledge generated in this round may be problematic. Considering the above situation, the quality type of the fragment cited by the LLM generated content in the retrieval knowledge can also be added in the reflection evaluation conclusion, so that when the quality of the fragment cited by the LLM generated content in the retrieval knowledge does not meet the requirements, the extraction data can be selected according to the quality type. At this time, the reflection evaluation conclusion can not only indicate the quality of the fragment cited by the LLM generated content in the retrieval knowledge, but also indicate the quality type of the fragment cited by the LLM generated content in the retrieval knowledge. The quality type can include: error or not detailed, etc. When the quality type is error, the information entropy gain algorithm can be used for retrieval knowledge generation, that is, the token information gain scores between the candidate sentences (i.e., Top M sentences) and the cited fragment with quality not meeting the requirements in the last round of retrieval knowledge are calculated, and the Top K sentences with the highest scores are retained. This process extracts the sentences with low relevance to the incorrect cited fragment recorded in the last round of retrieval knowledge from the candidate sentences, thereby correcting the invalid information gain of the retrieval knowledge. When the quality type is not detailed, the information completeness algorithm can be used for retrieval knowledge generation, that is, the semantic relevance scores between each sentence in the candidate sentences and the cited fragment with quality not meeting the requirements in the last round of retrieval knowledge are calculated, and the Top K sentences with the highest scores except the cited fragment with quality not meeting the requirements in the last round of retrieval knowledge are retained.This process is to extract more detailed details from the candidate sentences that are highly relevant to the not detailed reference fragments recorded in the retrieval knowledge of the last round and have different content, that is, to extract sentences that are strongly related to the not detailed reference fragments recorded in the retrieval knowledge of the last round and have more abundant content, so as to optimize the effective information gain of the retrieval knowledge of this round. In addition, the quality type can also be: correct, at which time it indicates that the quality of the fragment cited in the retrieval knowledge by the LLM generated content meets the requirements.

[0070] The prompt generation module 326 is mainly used for assembling the retrieval knowledge output by the dynamic chunking module 325 and the query text output by the query understanding module 323 into a prompt, and inputting the prompt to the LLM 327. For example, the prompt generation module 326 can add the retrieval knowledge and the query text to a preset prompt template to obtain the prompt.

[0071] The LLM 327 is mainly used for reasoning based on the prompt generated from the prompt generation module 326 to generate response text, that is, to obtain the generated content (also can be called "reasoning content" or "LLM generated content") related to the query text.

[0072] The reflection evaluation module 328 is mainly used to perform reflection evaluation on the quality of the retrieved knowledge and the LLM generated content in the current round according to the retrieved knowledge in the current round, the LLM generated content and the feedback of the user on the LLM generated content in the current round (i.e., the user feedback), to form an optimization basis for the retrieved knowledge, and to guide the generation of the retrieved knowledge in the next round. Specifically, as shown in FIG. 4, at S401, the fuzzy matching algorithm is used to process the retrieved knowledge and the LLM generated content to identify the segment (which can also be referred to as a "quoted segment") in the retrieved knowledge that is quoted by the LLM generated content. For example, the co-occurring segment between the retrieved knowledge and the LLM generated content can be calculated, and when the length of the co-occurring characters exceeds a set threshold, the segment is the quoted segment. At S402, the quoted segment is evaluated based on the feedback of the user on the LLM generated content (i.e., the user feedback) to obtain a reflection evaluation conclusion. For example, the pre-trained reflection evaluation classification model can be used to process the quoted segment and the user feedback to obtain the reflection evaluation conclusion of the retrieved knowledge. The reflection evaluation conclusion can indicate the quality of the segment in the retrieved knowledge that is quoted by the LLM generated content. Of course, the reflection evaluation conclusion can also indicate the quality type of the segment in the retrieved knowledge that is quoted by the LLM generated content, such as: incorrect, not detailed or correct, etc. As a possible implementation manner, the format of the reflection evaluation conclusion can be, but is not limited to, <reflection evaluation signal value sign, quoted segment text ref>. The reflection evaluation signal value can include any one of incorrect retrieved knowledge, not detailed retrieved knowledge and correct retrieved knowledge. In some embodiments, when the reflection evaluation conclusion indicates that the quality of the segment in the retrieved knowledge that is quoted by the LLM generated content meets the requirements (i.e., the retrieved knowledge in the current round is correct), it means that the retrieved knowledge in the current round meets the user query requirements, and therefore, the current session can be ended, and the retrieved knowledge in the current round can be stored in the knowledge base 322 as a new knowledge document, i.e., the retrieved knowledge is deposited, so that when a similar query is encountered, the knowledge document that meets the requirements can be quickly retrieved from the knowledge base, improving the retrieval efficiency. Because the dynamic chunking module 325 further dynamically extracts the top K sentences from the retrieved candidate knowledge data according to the user query and the reflection evaluation conclusion to assemble the retrieved knowledge, the retrieved knowledge stored in the knowledge base is likely to be different from the original knowledge data content, and therefore, storing the retrieved knowledge that meets the requirements in the knowledge base can achieve the purpose of depositing the useful retrieved knowledge in the dialogue process. Of course, the retrieved knowledge can also be deposited, and the specific implementation can be determined according to actual conditions, which is not limited here. In some embodiments, the user feedback can be the query text input by the user in the next round, or the direct feedback of the user on the LLM generated content (for example, the user can directly select the quality of the LLM generated content), or a combination of the two, etc., which can be determined according to actual conditions, and is not limited here.

[0073] In some embodiments, the architecture of the intelligent question-answering system 300 described above can also be as shown in Figure 5. As shown in Figure 5, the intelligent question-answering system 300 can be divided into a software part, a storage part, and a hardware part. The software part mainly consists of a knowledge base construction module 321, a query understanding module 323, a retrieval module 324, a dynamic chunking module 325, a prompt generation module 326, an LLM 327, and a reflection and evaluation module 328. The storage part mainly consists of a knowledge base 322, an inference device, and a training device. The inference device is mainly used to perform inference using the LLM 327, and it is the service device required for online inference. The training device is mainly used to train the LLM 327, and it is the service device required for offline training. The hardware part mainly consists of a client 310, a server (or server cluster), and a graphics card. The server (or server cluster) is mainly the physical machine used to deploy the various modules of the system. The graphics card is mainly the graphics card resources used for training and deploying the models involved in the system (such as the reflection and evaluation module 328).

[0074] The following section uses the example of storing data at the document level in the knowledge base to introduce the workflow of the intelligent question-answering system 300. As shown in Figure 6, the workflow of the intelligent question-answering system may include:

[0075] S601, Knowledge Base Construction. In this step, the original knowledge documents can be stored at the document level in a knowledge base such as an inverted index or a vector index library. First, the user needs to prepare a set of knowledge documents, each knowledge document being a text file. This set of knowledge documents is then stored in an inverted index (such as Elasticsearch) or a vector index (such as Faiss), effectively building the knowledge base index offline. Additionally, when the intelligent question-answering system evaluates the retrieved knowledge and confirms it meets the user's query requirements, the retrieved knowledge can be stored in the knowledge base index.

[0076] S602, Query Understanding. In this step, the user's original query can be rewritten or expanded, transforming it into an optimized query text. Specifically, the system first receives the user's query text, which can be the content of the user's first-round query or a query from a multi-round query during the dialogue. When it is a query from a multi-round query during the dialogue, the user query can be rewritten based on historical questions and answers and the current user query, mainly by supplementing necessary information mentioned in the context, eliminating ambiguity, and making the content more fluent.

[0077] S603, Retrieval. In this step, the N knowledge documents with the highest relevance scores to the optimized query text can be retrieved from the knowledge base.

[0078] S604, dynamic chunking. In this step, if there is no conclusion of the reflection evaluation of the last round, the semantic relevance algorithm can be used to generate the retrieved knowledge, i.e., the vector semantic relevance scores (i.e., vector inner product) of each sentence in the optimized query text and the candidate document are calculated, the Top K sentences with the highest scores are selected, and the retrieved knowledge of this round is assembled. If the conclusion of the reflection evaluation of the last round is provided, the sentences in the candidate document that meet the conditions can be extracted based on the optimized query text and the conclusion of the reflection evaluation of the last round, and the retrieved knowledge of this round is assembled (for details, see the description of the dynamic chunking module 325 in the foregoing FIG. 3, which will not be described here again). In some embodiments, the acceptable input length of the LLM used to generate the response is L, and the total length of the Top K sentences does not exceed L, so that all the contents of the retrieved knowledge can be input into the LLM.

[0079] S605, generating a response. In this step, the retrieved knowledge of this round and the optimized query text can be assembled into a Prompt and input into the LLM to generate a response text (i.e., LLM generated content).

[0080] S606, displaying the response result and the guidance information. In this step, the server in the system can send the response result and the guidance information to the client for use by the user. Specifically, it includes the LLM generated content, the document link of the cited retrieved knowledge (optional), and the guidance information for guiding the user to provide feedback evaluation (optional).

[0081] S607, reflection evaluation. In this step, the quality of the retrieved knowledge and the LLM generated content can be evaluated based on the retrieved knowledge, the LLM generated content, and the user feedback (such as the user's next round of follow-up query text and / or the user's feedback on the LLM generated content). The user's next round of query can provide feedback content using the guidance information or can provide the next round of follow-up query without using the guidance information. For the reflection evaluation process, see the description of the reflection evaluation module 328 in the foregoing FIG. 3, which will not be described here again.

[0082] After introducing the workflow of the intelligent question answering system 300, the product implementation form of the software modules thereof will be introduced below.

[0083] For example, FIG. 7 shows a product implementation schematic diagram of software modules in an intelligent question answering system according to an embodiment of the present application. The implementation of the software modules in the server 320 of the intelligent question answering system 300 can be, but is not limited to, program codes contained in a machine learning and deep learning platform software and deployed on a server hardware. The program codes can exist, but are not limited to, inside a runtime engine, a memory management module and a communication management module of the platform software, and outside the existing modules, etc. In addition, the program codes can also run in the host memory and / or the graphics processing unit (GPU) memory of the server. As shown in FIG. 7, the query understanding module, the retrieval module, the dynamic chunking module, the knowledge base construction module, the reflection evaluation module and the response generation module in the intelligent question answering system 300 can all be program codes deployed on a server hardware. At the same time, the product implementation of the intelligent question answering system 300 can have query entry, result viewing and guided feedback, etc. User interaction content. Through the query entry, the user can input query text content in the dialogue input box. The content can be the first round of query to initiate a question and answer, or multiple rounds of query (i.e. follow-up) in the question and answer process. Through the result viewing, the user can receive the response result provided by the system, can view the answer text content of the LLM, and the knowledge documents cited by the current retrieval, etc. Through the guided feedback, the user can receive the response result provided by the system, and also receive the guidance information provided by the system. The guidance information is the evaluation feedback of the user on the cited retrieval knowledge and the LLM response result. The user can selectively use the guidance text as the query text for the next round of dialogue, or can use other queries for the next round of follow-up. Whether the user uses the guidance information text for feedback or not, the dialogue can be carried out through the dialogue input box.

[0084] The flow and operation of data in the intelligent question answering system 300 shown in FIG. 3 are described below in the manner of the technical concept shown in FIG. 2.

[0085] Referring to FIG. 8, after the user initiates a query, the user query can be optimized to obtain query text. After obtaining the query text, retrieval of knowledge data (i.e., knowledge documents and / or knowledge fragments) and / or retrieval knowledge previously deposited in the knowledge base can be performed in the knowledge base. Then, in the dynamic chunking part, in combination with the user query of this round and the reflection evaluation conclusion of the generated retrieval knowledge of the previous round of dialogue in this session, the Top K sentences that meet the requirements are extracted from the retrieved knowledge data and / or retrieval knowledge, and the retrieval knowledge required for this round of dialogue is generated. Then, through the retrieval knowledge generated in this round, LLM generated content is obtained and displayed to the user. After the user provides feedback on the LLM generated content of this round, reflection evaluation can be performed on the retrieval knowledge and LLM generated content generated in this round of dialogue in combination with the user feedback to obtain a reflection evaluation conclusion. When the reflection evaluation conclusion indicates that the quality of the fragment cited by the LLM generated content in the retrieval knowledge generated in this round of dialogue meets the requirements, the retrieval knowledge generated in this round of dialogue can be stored in the knowledge base, and the session can be ended. When the reflection evaluation conclusion indicates that the quality of the fragment cited by the LLM generated content in the retrieval knowledge generated in this round of dialogue does not meet the requirements, the reflection evaluation conclusion obtained in this round of dialogue can be used as the optimization basis for retrieval knowledge generation in the next round of dialogue. It should be understood that, for the reflection evaluation of the retrieval knowledge, the actual situation can be configured, which is not limited here.

[0086] In order to facilitate understanding of the intelligent question and answer system described above, the operation of the intelligent question and answer system is introduced below.

[0087] Referring to FIG. 9, FIG. 9 shows an interface provided by the client during a session. As shown in FIG. 9, after the user inputs query text in the dialogue input box of the system interface of each round of dialogue, the system provides an interface of LLM response results, cited retrieval knowledge links, and guide information. In FIG. 9, four rounds of dialogue are performed between the user and the system. The retrieval knowledge cited in the first round of dialogue is incorrect and cannot help the LLM generate correct response results, so the user asks a follow-up question in the second round of dialogue, and the system optimizes the retrieval knowledge based on the follow-up query. The retrieval knowledge cited in the second round of dialogue is not detailed and cannot help the LLM generate correct response results, so the user asks a follow-up question in the third round of dialogue, and the system optimizes the retrieval knowledge based on the follow-up query. The retrieval knowledge cited in the third round of dialogue is correct and can help the LLM generate complete response results to meet the user's needs, so the user confirms, and then the system summarizes and displays the retrieval knowledge that meets the user's query, and stores it in the knowledge base. The operation of the system in the four rounds of dialogue is as follows:

[0088] 1) Query understanding: the system receives the user's query text, inputs the query text into the open source LLM in the first round of dialogue, and lets the LLM output the rewritten original user query, and outputs the optimized query text. For example, the original query text is "Python code for matching enterprise mailbox using regular expression", and the optimized query text can be "Python code for matching enterprise mailbox using regular expression".

[0089] 2) Retrieval: the system recalls N knowledge data with the highest relevance score from the pre-constructed knowledge base to the optimized query.

[0090] 3) Dynamic chunking: the relevance score of the optimized query text and each sentence in the candidate knowledge data is calculated in turn, and the top K sentences with the highest relevance score are selected. The total length of the selected sentences does not exceed the input threshold of the LLM used for reasoning. The first round of retrieval knowledge is from the document "Python regular expressions".

[0091] 4) Generating responses: the system splices the optimized query text and the retrieval knowledge obtained from the dynamic chunking step to assemble a prompt, which is input to the LLM used for reasoning to obtain a response text (i.e. LLM generated content).

[0092] 5) Response result and guidance information display: the system sends the LLM generated response text, the document link of the cited retrieval knowledge "Python regular expressions", and the guidance information text "Is the cited document and answer helpful? Please continue to ask questions if the answer is not correct or detailed enough." to the client for user use.

[0093] 6) Reflection evaluation: Based on the retrieved knowledge, the LLM generated response, and the second round of follow-up query text "The email format is incorrect, and the code coverage format is too broad", the quality of the retrieved knowledge and the generated response is evaluated. Specifically, the reflection evaluation is divided into 3 sub-steps: (1) Citation source detection: Find the citation fragment between the retrieved knowledge and the large model response result through fuzzy matching algorithm, that is, calculate the co-occurrence fragment between the retrieved knowledge and the response text, when the co-occurrence character length exceeds the set threshold, the fragment is the citation source. The citation fragment content of the first round of dialogue is "pattern = re.compile(r'^[a-zA-Z0-9._-]+@[a-zA-Z0-9-]+\.[a-zA-Z.]{2,18}$')". (2) Citation source reflection evaluation: The reflection evaluation classification model receives two inputs: user feedback "The email format is incorrect, and the code coverage format is too broad", and the citation fragment, to make a quality judgment on the citation source. The reflection evaluation model here can be a Chatglm3 model, which outputs the reflection evaluation classification signal (i.e. classification result): "The retrieved knowledge is incorrect". The reflection evaluation signal and the citation source fragment are recorded as the reflection evaluation conclusion, that is, <The retrieved knowledge is incorrect, pattern = re.compile(r'^[a-zA-Z0-9._-]+@[a-zA-Z0-9-]+\.[a-zA-Z.]{2,18}$')>. ^ [a-zA-Z0-9._-]+@[a-zA-Z0-9-]+\.[a-zA-Z.]{2,18}$')>.

[0094] 7) Query understanding: During the second round of dialogue, the query text of this round is spliced with the query and large model answer content of the first round, and input into the open source LLM, which outputs the rewritten original user query, and outputs the optimized query text. For example: the original query "The code coverage format is too broad", the optimized query "Use regular expression to match enterprise email in python code".

[0095] 8) Retrieval: Retrieve the N knowledge data with the highest relevance score from the knowledge base according to the optimized query text.

[0096] 9) Dynamic chunking: Based on the conclusion of the first round of reflection evaluation, extract the sentences in the candidate knowledge data that meet the conditions, the current reflection evaluation conclusion is <search knowledge is wrong, pattern = re.compile(r'^[a-zA-Z0-9._-]+@[a-zA-Z0-9-]+\.[a-zA-Z.]{2,18}$')>, then the system generates search knowledge based on the information entropy gain algorithm. Specifically, first, calculate the relevance score of each sentence in the optimized query and the candidate document in turn, select the top 2*TopK sentences with the highest relevance score, and then calculate the token information gain score between each sentence and the reference fragment, and keep the top K sentences with the highest score as the dynamic chunking. The length of the selected sentences does not exceed the input threshold of the large model used for reasoning. The search knowledge of the second round is from the document <Regular Expression (II) Common Regular Expressions - Validate Email>.

[0097] 10) Generate response: The system splices the optimized query text and the search knowledge obtained from the dynamic chunking step to assemble a Prompt, which is input to the LLM used for reasoning to obtain the response text.

[0098] 11) Response result and guidance information display: The system sends the LLM's generated response text, the document link of the cited search knowledge <Regular Expression (II) Common Regular Expressions - Validate Email>, and the guidance information text "Is the cited document and answer helpful? Please continue to ask questions with the answer or XX is wrong, XX is not detailed." to the user for use.

[0099] 12) Reflection evaluation: After the user inputs "The answer is not detailed, what are the rules followed by the beginning of the expression" in the third round of dialogue, based on the retrieved knowledge, the LLM generates a response, and according to the user's next round of follow-up query text "The answer is not detailed, what are the rules followed by the beginning of the expression" provided by the guide information, the quality of the retrieved knowledge and the generated response is reflected and evaluated. Specifically, the reflection evaluation is divided into 3 sub-steps: (1) Citation source detection: Find the citation fragment between the retrieved knowledge and the LLM response result by fuzzy matching algorithm, that is, calculate the co-occurrence fragment between the retrieved knowledge and the response text, when the length of co-occurrence characters exceeds the set threshold, the fragment is the citation source fragment. The content of the citation fragment in the second round of dialogue is "let reg = ^\w+((-\w+)|(\.w+))*@[A-Za-z0-9]+((\.|-)[A-Za-z0-9]+)*\.[a-zA-Z0-9]+$ / ".(2) Citation source reflection evaluation: The reflection evaluation classification model receives two inputs: user feedback "The answer is not detailed, what are the rules followed by the beginning of the expression" and citation fragment text, to judge the quality of the citation source, the reflection evaluation model here can be a Chatglm3 model, using the model to get the reflection evaluation classification signal (i.e. classification result): "The retrieved knowledge is not detailed". Then record the reflection evaluation conclusion, that is, <The retrieved knowledge is not detailed, let reg = ^\w+((-\w+)|(\.w+))*@[A-Za-z0-9]+((\.|-)[A-Za-z0-9]+)*\.[a-zA-Z0-9]+$ / >.

[0100] 13) Query understanding: During the third round of dialogue, the query text of this round is spliced with the query text of the previous round and the LLM answer content, and input into the open source LLM, so that the LLM outputs the rewritten original user query, and outputs the optimized query text. For example: the original query "The answer is not detailed, what are the rules followed by the beginning of the expression", the optimized query "What are the rules followed by the regular expression of the python code matching the enterprise mailbox".

[0101] 14) Retrieval: Recall the N knowledge data with the highest relevance score from the knowledge base to the optimized query text.

[0102] 15) Dynamic chunking: Based on the optimized query text and the conclusion of the second round of reflection evaluation, extract sentences from the candidate knowledge data to form the retrieval knowledge. The current reflection evaluation conclusion is <the retrieval knowledge is not detailed, let reg = ^\w+((-\w+)|(\.w+))*@[A-Za-z0-9]+((\.|-)[A-Za-z0-9]+)*\.[a-zA-Z0-9]+$ / >, then the system generates retrieval knowledge based on the content completeness algorithm. Specifically, first, calculate the relevance score of the optimized query text and each sentence in the candidate knowledge data in turn, select the top 2*TopK sentences with the highest relevance score, and then calculate the semantic relevance score between each sentence and the reference fragment, and keep the top K sentences with the highest score except the reference fragment as the retrieval knowledge. The length of the selected sentences does not exceed the input threshold of the LLM used for reasoning. The third round of retrieval knowledge is from the document <Regular Expression (II) Common Regular Expressions - Validate Email>.

[0103] 16) Generate response: splice the optimized query text and the retrieval knowledge obtained in the dynamic chunking step to form a Prompt, input it to the LLM, and get the response text.

[0104] 17) Response result and guidance information display: the system sends the LLM's generated response text, the document link of the cited retrieval knowledge <Regular Expression (II) Common Regular Expressions - Validate Email>, and the guidance information text "Is the cited document and answer helpful? Please continue to ask questions with the answered or XX is not correct, XX is not detailed." to the client for user use.

[0105] 18) Reflection evaluation: After the user inputs "end the dialogue, generate a knowledge summary" in the fourth round of dialogue, the quality of the retrieved knowledge and the generated response is evaluated based on the retrieved knowledge, the LLM-generated response, and the user's next round of follow-up query text "end the dialogue, generate a knowledge summary" provided according to the guidance information. Specifically, the reflection evaluation is divided into three sub-steps: (1) Citation source detection: Find the citation fragment between the retrieved knowledge and the LLM response result by fuzzy matching algorithm, i.e. calculate the co-occurrence fragment between the retrieved knowledge and the response text, when the length of the co-occurrence characters exceeds the set threshold, the fragment is the citation source fragment. The content of the citation fragment in the third round of dialogue is "Rule analysis 1. Start with a number-letter underscore (1 to multiple digits) 2. Also can be - number-letter underscore or. Number-letter underscore, overall zero to multiple times\nSummary => The name of the mailbox is composed of "numbers, letters, underscores, -, and." parts, but - / . cannot appear consecutively and cannot be the start\nThat is => \w+((-\w+)|(\.\w+))*". (2) Citation source reflection evaluation: The reflection evaluation classification model receives two inputs: user feedback "end the dialogue, generate a knowledge summary" and citation fragment, and makes a quality judgment on the citation source. The reflection evaluation model here can be a Chatglm3 model, which uses the model to get the reflection evaluation classification signal (i.e. classification result): "the retrieved knowledge is correct". When the reflection evaluation conclusion meets the user's query needs, the retrieved knowledge content is stored as a new knowledge document in the knowledge base.

[0106] 19) Response result and guidance information display: The system sends the retrieved knowledge stored in the knowledge base in the reflection evaluation step to the client, including the document link of "Regular Expression (II) Common Regular Expressions - Validate Email" and the content fragment cited by the LLM, for the user to use.

[0107] From the above description of the figure, it can be seen that in the multiple rounds of dialogue in a session, the quality of the retrieved knowledge in this round can be accurately evaluated by combining the retrieved knowledge in this round, the LLM-generated content in this round, and the user's feedback in this round. When the quality of the retrieved knowledge is poor, the quality of the LLM-generated content is likely to be poor, so the quality of the LLM-generated content can also be accurately evaluated. Further, by optimizing the retrieval of knowledge in the next round of dialogue based on the erroneous knowledge in the retrieved knowledge in the current round, the quality of the retrieved knowledge in the next round of dialogue can be improved, the LLM's answering ability can be greatly improved, and the number of rounds required for user questioning and answering can be reduced.

[0108] It should be understood that the fourth round of dialog in FIG. 9 can be selectively added, which is not limited here. In FIG. 9, the user can choose to confirm that the answer in the third round of dialog is correct, can directly make a new session in the session window, can close the session window, etc. Preferably, after a session ends, a knowledge summary of the session is displayed to the user to improve the user interaction experience. For generating the session knowledge summary, the user can actively trigger, the system can automatically trigger, which is not limited here. When the user actively triggers, the user can input a corresponding instruction in the dialog input box on the client, can select a certain control (such as control 91, etc.) displayed on the client, can trigger through a voice instruction, etc.

[0109] The above is the related introduction of the intelligent question answering system provided by the embodiments of the present application. Based on the above, the intelligent question answering method provided by the embodiments of the present application is introduced.

[0110] For example, FIG. 10 shows a flowchart of an intelligent question answering method provided by the embodiments of the present application. It can be understood that the method can be executed by any device, equipment, platform, device cluster with computing and processing capabilities. For example, the method can be executed by an intelligent question answering device, wherein the device can be realized by software and / or hardware, can be configured in an electronic device or a server, typically, can be configured on a cloud computing platform. For ease of description, the cloud computing platform will be taken as the execution subject for introduction. As shown in FIG. 10, the intelligent question answering method can include the following steps:

[0111] S1001, retrieving based on a first user query in the latest round of dialog in a first session to obtain first retrieval knowledge required in the latest round of dialog.

[0112] In this embodiment, the client associated with the cloud computing platform can transmit the query input by the user in each round of dialogue in a session to the cloud computing platform. After receiving the query input by the user in the latest round of dialogue, the cloud computing platform can retrieve the first retrieval knowledge required in the latest round of dialogue from the knowledge base based on the query input by the user in the latest round of dialogue. The knowledge base stores knowledge documents and / or knowledge segments. In some embodiments, when the latest round of dialogue is the first round of dialogue in the session, the cloud computing platform can directly retrieve the knowledge base based on the query input by the user, and extract at least one sentence with the highest relevance to the query input by the user from the retrieval result to obtain the first retrieval knowledge. When the latest round of dialogue is not the first round of dialogue in the session, the cloud computing platform can first retrieve the knowledge base based on the query input by the user in the latest round of dialogue to obtain knowledge related to the intent of the query input by the user. Then, the cloud computing platform extracts at least one sentence with the highest relevance to the query input by the user from the knowledge related to the intent of the query input by the user based on the second reference segment to filter out the first retrieval knowledge. The second reference segment is a segment of the retrieval knowledge required in the previous round of dialogue that is referenced by the response text in the previous round of dialogue, and the quality of the second reference segment does not meet the requirements. In addition, the first retrieval knowledge is weakly related to the second reference segment, or the first retrieval knowledge is strongly related to the second reference segment and has a higher content richness than the second reference segment. For example, when the quality type of the second reference segment is incorrect, the first retrieval knowledge is weakly related to the second reference segment, thereby achieving the purpose of invalid information gain of correcting the retrieval knowledge. When the quality type of the second reference segment is lack of content richness, the first retrieval knowledge is strongly related to the second reference segment and has a higher content richness than the second reference segment, thereby achieving the purpose of effective information gain of optimizing the retrieval knowledge. The process of obtaining the first retrieval knowledge can be found in the foregoing description of the retrieval module 324 and the dynamic segmentation module 325, and will not be repeated here.

[0113] S1002, processing the first retrieval knowledge and the first user query through the neural network model to obtain the first response text.

[0114] In this embodiment, after obtaining the first retrieval knowledge, the neural network model can be used to process the first retrieval knowledge and the first user query to obtain the first response text. For example, the first retrieval knowledge and the first user query can be added to the Prompt module to obtain a Prompt. Then, the Prompt is input into the neural network model for processing.

[0115] S1003, based on the first search knowledge, the first response text and the first feedback of the user to the first response text, performing reflection evaluation on the first search knowledge to obtain a first reflection evaluation conclusion, the first reflection evaluation conclusion being used to indicate the quality of a first cited fragment in the first search knowledge, the first cited fragment being a fragment cited by the first response text.

[0116] In this embodiment, after obtaining the first response text, the cloud computing platform can transmit the response text to the client associated therewith, so as to show the user the first response text through the client. After viewing the first response text, the user can make corresponding feedback. For example, the user can ask follow-up questions in the next round of dialogue of the current session, or input a user query in the first round of dialogue of a new session, and the like. Then, the client can transmit the feedback of the user to the first response text (i.e., the first feedback) to the cloud computing platform.

[0117] After receiving the first feedback, the cloud computing platform can perform reflection evaluation on the first search knowledge based on the first search knowledge, the first response text and the first feedback of the user to the first response text, to obtain a first reflection evaluation conclusion. The first reflection evaluation conclusion can be used to indicate the quality of a first cited fragment in the first search knowledge. The first cited fragment is a fragment cited by the first response text. In this way, the quality of the search knowledge retrieved from the knowledge base in the latest round of dialogue can be known. In some embodiments, the cloud computing platform can first calculate a co-occurrence fragment between the first search knowledge and the first response text to obtain the first cited fragment. Then, the cloud computing platform can process the first cited fragment and the first feedback through a reflection evaluation model to obtain the first reflection evaluation conclusion. For the process of obtaining the first reflection evaluation conclusion, reference can be made to the foregoing description of the reflection evaluation module 328, which will not be repeated here.

[0118] S1004, in a case where the quality of the first cited fragment does not meet the requirement, performing optimization on second search knowledge required in a next round of dialogue based on the first cited fragment, wherein the second search knowledge is obtained based on the first cited fragment and a second user query in the next round of dialogue.

[0119] In this embodiment, when the quality of the first cited fragment does not meet the requirement, the first search knowledge cannot help the neural network model to generate a correct response text. Therefore, the cloud computing platform can perform optimization on second search knowledge required in a next round of dialogue based on the first cited fragment, so as to help the neural network model to generate a correct response text. The second search knowledge is obtained based on the first cited fragment and a second user query in the next round of dialogue. For the process of optimizing the second search knowledge, reference can be made to the foregoing description of the retrieval module 324 and the dynamic partitioning module 325, which will not be repeated here.

[0120] S1005, in the case where the quality of the first cited segment meets the requirement, one or more of the following is performed: storing the first retrieved knowledge, generating a knowledge summary of the first session, or ending the first session.

[0121] In the embodiment, in the case where the quality of the first cited segment does not meet the requirement, it indicates that the response text generated by the neural network model meets the user demand, and at this time the cloud computing platform can perform one or more of the following: storing the first retrieved knowledge, generating a knowledge summary of the current session, or ending the current session. After storing the first retrieved knowledge, accurate answers can be quickly found when similar problems are encountered subsequently, and the retrieval efficiency is improved. Generating a knowledge summary of the current session can facilitate the user to understand the content of the current session, and improve the user interaction experience.

[0122] In this way, in the intelligent question and answer process, the quality of the retrieved knowledge generated for each round of conversation is evaluated, and the retrieved knowledge with poor quality is corrected in the next round of conversation, so that the incorrect knowledge in the retrieved knowledge can be corrected in time, the probability of introducing incorrect knowledge is reduced, and the answer quality of the neural network model is improved.

[0123] It can be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, the technical features involved in each of the above-described embodiments or embodiments can be combined according to the actual situation, and the combined scheme is still within the protection scope of the present application.

[0124] Based on the method in the above embodiment, the present embodiment further provides an intelligent question and answer device.

[0125] Exemplarily, FIG. 11 shows a structural schematic diagram of an intelligent question-answering apparatus provided by an embodiment of the present application. As shown in FIG. 11, the intelligent question-answering apparatus 1100 comprises a retrieval module 1101 and a processing module 1102. The retrieval module 1101 is configured to perform retrieval based on a first user query in a latest round of dialogue in a first session, to obtain first retrieval knowledge required in the latest round of dialogue. The processing module 1102 is configured to perform processing on the first retrieval knowledge and the first user query by using a neural network model, to obtain a first response text. The processing module 1102 is further configured to perform reflection evaluation on the first retrieval knowledge based on the first retrieval knowledge, the first response text and first feedback of the user on the first response text, to obtain a first reflection evaluation conclusion, the first reflection evaluation conclusion being used to indicate a quality of a first cited segment in the first retrieval knowledge, the first cited segment being a segment cited by the first response text. The processing module 1102 is further configured to perform optimization on second retrieval knowledge required in a next round of dialogue based on the first cited segment in a case where the quality of the first cited segment does not meet a requirement, wherein the second retrieval knowledge is obtained by performing retrieval based on the first cited segment and a second user query in the next round of dialogue.

[0126] In some embodiments, the latest round of dialogue is a non-first round of dialogue in the current session. At this time, when the retrieval module 1101 performs retrieval based on the first user query in the latest round of dialogue to obtain the first retrieval knowledge required in the latest round of dialogue, the retrieval module 1101 is specifically configured to: perform retrieval in a knowledge base based on the first user query, to obtain knowledge related to an intention of the user query, wherein the knowledge base stores knowledge documents and / or knowledge segments; and select the first retrieval knowledge from the knowledge related to the intention of the user query based on a second cited segment, the first retrieval knowledge being weakly related to the second cited segment, or the first retrieval knowledge being strongly related to the second cited segment and having a content richness higher than that of the second cited segment. The second cited segment is a segment cited by a response text in a previous round of dialogue in the first retrieval knowledge required in the previous round of dialogue, and the quality of the second cited segment does not meet the requirement.

[0127] In some embodiments, in a case where the quality type of the second cited segment is erroneous, the first retrieval knowledge is weakly related to the second cited segment; and in a case where the quality type of the second cited segment is a lack of content richness, the first retrieval knowledge is strongly related to the second cited segment and has a content richness higher than that of the second cited segment.

[0128] In some embodiments, the processing module 1102 is further configured to: in a case where the quality of the first cited segment meets the requirement, perform one or more of the following: store the first retrieval knowledge; generate a knowledge summary of the first session; or end the first session.

[0129] In some embodiments, the processing module 1102 is specifically configured to: calculate a co-occurrence segment between the first retrieved knowledge and the first response text to obtain a first reference segment, when performing the reflection evaluation on the first retrieved knowledge based on the first retrieved knowledge, the first response text, and the first feedback of the user on the first response text to obtain a first reflection evaluation conclusion.

[0130] In some embodiments, the first feedback is a follow-up question of the user in the next round of conversation.

[0131] In some embodiments, the retrieval module 1101 and the processing module 1102 shown in FIG. 11 can be implemented by software or by hardware. For example, the retrieval module 1101 is taken as an example to introduce the implementation of the retrieval module 1101. Similarly, the implementation of the processing module 1102 can refer to the implementation of the retrieval module 1101.

[0132] As an example of the module as a software functional unit, the retrieval module 1101 can include code running on a computing instance. The computing instance can include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance can be one or more. For example, the retrieval module 1101 can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed in the same region, or can be distributed in different regions. Further, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same availability zone (AZ), or can be distributed in different AZs, each AZ including one data center or multiple data centers with similar geographical locations. Generally, one region can include multiple AZs.

[0133] Similarly, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC), or can be distributed in multiple VPCs. Generally, one VPC is set in one region, and communication between two VPCs in the same region and between VPCs in different regions needs to be set in each VPC to realize the interconnection between VPCs through a communication gateway.

[0134] As an example of a hardware functional unit, the retrieval module 1101 can include at least one computing device, such as a server or the like. Alternatively, the retrieval module 1101 can also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), and the like. The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0135] The plurality of computing devices included in the retrieval module 1101 can be distributed in the same region or in different regions. The plurality of computing devices included in the retrieval module 1101 can be distributed in the same AZ or in different AZs. Similarly, the plurality of computing devices included in the retrieval module 1101 can be distributed in the same VPC or in multiple VPCs. The plurality of computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0136] It should be noted that, in other embodiments, the retrieval module 1101 can be configured to perform any of the steps of the intelligent question answering method described in the above embodiments, and the processing module 1102 can be configured to perform any of the steps of the intelligent question answering method described in the above embodiments. In addition, the steps implemented by the retrieval module 1101 and the processing module 1102 can be specified as needed, and the retrieval module 1101 and the processing module 1102 can be configured to implement different steps of the intelligent question answering method described in the above embodiments to implement all the functions of the intelligent question answering device 1100 shown in FIG. 11.

[0137] The present application also provides a computing device 1200. As shown in FIG. 12, the computing device 1200 includes a bus 1202, a processor 1204, a memory 1206, and a communication interface 1208. The processor 1204, the memory 1206, and the communication interface 1208 communicate with each other through the bus 1202. The computing device 1200 can be a server or an electronic device. It should be understood that the present application does not limit the number of processors and memories in the computing device 1200.

[0138] The bus 1202 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is represented in FIG. 12, but it does not mean that there is only one bus or only one type of bus. The bus 1204 can include a path for transmitting information between various components (e.g., the memory 1206, the processor 1204, the communication interface 1208) of the computing device 1200.

[0139] The processor 1204 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0140] The memory 1206 can include a volatile memory (e.g., a random access memory (RAM)) and a non-volatile memory (e.g., a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD)).

[0141] The memory 1206 stores executable program codes, and the processor 1204 executes the executable program codes to respectively implement the functions of the retrieval module 1101 and the processing module 1102 shown in FIG. 11, thereby implementing the intelligent question answering method described in the above embodiments. That is, the memory 1206 stores instructions for executing the intelligent question answering method described in the above embodiments.

[0142] Alternatively, the memory 1206 stores executable program codes, and the processor 1204 executes the executable program codes to respectively implement the functions of the intelligent question answering apparatus 1100 shown in FIG. 11, thereby implementing the intelligent question answering method described in the above embodiments. That is, the memory 1206 stores instructions for executing the intelligent question answering method described in the above embodiments.

[0143] The communication interface 1203 uses a transceiving module such as, but not limited to, a network interface card, a transceiver, and the like to enable communication between the computing device 1200 and other devices or communication networks.

[0144] The embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center.

[0145] As shown in FIG. 13, the computing device cluster includes at least one computing device 1200. The memory 1206 in one or more computing devices 1200 in the computing device cluster can store the same instructions for performing the intelligent question answering method described in the above embodiments.

[0146] In some possible implementations, the memory 1206 in one or more computing devices 1200 in the computing device cluster can also respectively store partial instructions for performing the intelligent question answering method described in the above embodiments. In other words, the combination of one or more computing devices 1200 can collectively execute the instructions for performing the intelligent question answering method described in the above embodiments.

[0147] It should be noted that the memory 1206 in different computing devices 1200 in the computing device cluster can store different instructions for respectively performing part of the functions of the intelligent question answering apparatus 1100 shown in FIG. 11. That is, the instructions stored in the memory 1206 in different computing devices 1200 can implement the functions of one or more of the retrieval module 1101 and the processing module 1102.

[0148] In some possible implementations, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network, a local area network, or the like. FIG. 14 shows one possible implementation. As shown in FIG. 14, two computing devices 1200A and 1200B are connected through a network. Specifically, the communication interface in each computing device is connected to the network. In this type of possible implementation, the memory 1206 in the computing device 1200A stores instructions for performing the functions of the retrieval module 1101. Meanwhile, the memory 1206 in the computing device 1200B stores instructions for performing the functions of the processing module 1102.

[0149] It should be understood that the functions of the computing device 1200A shown in FIG. 14 can also be completed by multiple computing devices 1200. Similarly, the functions of the computing device 1200B can also be completed by multiple computing devices 1200.

[0150] The embodiments of the present application also provide another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to the connection manner of the computing device cluster described with reference to FIG. 13 and FIG. 14. The difference is that the same instruction for performing the method in the above embodiments can be stored in the memory 1206 of one or more computing devices 1200 in the computing device cluster.

[0151] In some possible implementation manners, part of the instruction for performing the method can also be stored in the memory 1206 of one or more computing devices 1200 in the computing device cluster respectively. In other words, the combination of one or more computing devices 1200 can collectively execute the instruction for performing the method.

[0152] Based on the method in the above embodiments, the embodiments of the present application provide a computer readable storage medium including computer program instructions, when the computer program instructions are executed by a computing device, the computing device executes the method in the above embodiments; or when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method in the above embodiments. For example, the computer readable storage medium can be any available medium or data storage device including one or more available media that a computing device can store. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk) and the like.

[0153] Based on the method in the above embodiments, the embodiments of the present application provide a computer program product including instructions, when the instructions are run by a computing device, the computing device executes the method in the above embodiments, or when the instructions are run by a computing device cluster, the computing device cluster executes the method in the above embodiments.

[0154] It can be understood that the processor in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0155] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to a processor, so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0156] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted by the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, cloud computing platform, or data center to another website, computer, cloud computing platform, or data center through a wired (such as a coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a cloud computing platform, data center, etc. that includes one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0157] It can be understood that the various numerical numbers involved in the embodiments of the present application are only for convenient differentiation, and do not limit the scope of the embodiments of the present application.

[0158] Finally, it should be noted that: the above examples are used to illustrate the technical solutions of the present application, but not limited to them; although the present application is described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. An intelligent question answering method, characterized by, The method comprises: retrieving, based on a first user query in a latest round of dialogue in a first session, first retrieval knowledge required in the latest round of dialogue; processing, by a neural network model, the first retrieval knowledge and the first user query to obtain first response text; reflectively evaluating the first retrieval knowledge based on the first retrieval knowledge, the first response text, and first feedback of a user on the first response text to obtain a first reflective evaluation conclusion, the first reflective evaluation conclusion being used to indicate quality of a first cited segment in the first retrieval knowledge, the first cited segment being a segment cited by the first response text; in a case where the quality of the first cited segment does not meet requirements, optimizing, based on the first cited segment, second retrieval knowledge required in a next round of dialogue, wherein the second retrieval knowledge is obtained by retrieval based on the first cited segment and a second user query in the next round of dialogue.

2. The method of claim 1, wherein, The latest round of dialogue is a non-first round of dialogue in the first session. The retrieving, based on a first user query in a latest round of dialogue in a first session, first retrieval knowledge required in the latest round of dialogue comprises: retrieving, based on the first user query, knowledge related to a user query intention from a knowledge base, wherein the knowledge base stores knowledge documents and / or knowledge segments; screening, based on a second cited segment, the first retrieval knowledge from the knowledge related to the user query intention, the first retrieval knowledge being weakly related to the second cited segment, or the first retrieval knowledge being strongly related to the second cited segment and having higher content richness than the second cited segment; wherein the second cited segment is a segment cited by response text in a previous round of dialogue from retrieval knowledge required in the previous round of dialogue, and the quality of the second cited segment does not meet requirements.

3. The method of claim 2, wherein, In a case where the quality type of the second cited segment is incorrect, the first retrieval knowledge is weakly related to the second cited segment. In a case where the quality type of the second cited segment is a lack of content richness, the first retrieval knowledge is strongly related to the second cited segment and has higher content richness than the second cited segment.

4. The method according to any of claims 1 to 3, characterized in that, The method further comprises: in a case where the quality of the first cited segment meets requirements, performing one or more of the following: storing the first retrieval knowledge; generating a knowledge summary of the first session; or, ending the first session.

5. The method according to any of claims 1 to 4, characterized in that, The reflectively evaluating the first retrieval knowledge based on the first retrieval knowledge, the first response text, and first feedback of a user on the first response text to obtain a first reflective evaluation conclusion comprises: calculating a co-occurrence segment between the first retrieval knowledge and the first response text to obtain the first cited segment; processing, by a reflective evaluation model, the first cited segment and the first feedback to obtain the first reflective evaluation conclusion.

6. The method according to any one of claims 1 to 5, characterized in that, The first feedback is a follow-up question of a user in the next round of dialogue.

7. An intelligent question answering apparatus, characterized by comprising: Comprise: The retrieval module performs retrieval based on a first user query in a latest round of dialogue in a first session to obtain first retrieval knowledge required in the latest round of dialogue; The processing module performs processing on the first retrieval knowledge and the first user query by using a neural network model to obtain first response text; The processing module further performs reflection evaluation on the first retrieval knowledge based on the first retrieval knowledge, the first response text, and first feedback of a user on the first response text to obtain a first reflection evaluation conclusion, which indicates quality of a first cited segment in the first retrieval knowledge, the first cited segment being a segment cited by the first response text; When the quality of the first cited segment does not meet requirements, the processing module performs optimization on second retrieval knowledge required in a next round of dialogue based on the first cited segment, wherein the second retrieval knowledge is obtained based on the first cited segment and a second user query in the next round of dialogue.

8. The apparatus of claim 7, wherein, The latest round of dialogue is a non-first round of dialogue in the current session; When the retrieval module performs retrieval based on the first user query in the latest round of dialogue to obtain the first retrieval knowledge required in the latest round of dialogue, the retrieval module specifically performs the following operations: performing retrieval in a knowledge base based on the first user query to obtain knowledge related to a user query intention, wherein the knowledge base stores knowledge documents and / or knowledge segments; filtering the first retrieval knowledge from the knowledge related to the user query intention based on a second cited segment, the first retrieval knowledge being weakly related to the second cited segment, or the first retrieval knowledge being strongly related to the second cited segment and having higher content richness than the second cited segment; The second cited segment is a segment in retrieval knowledge required in a previous round of dialogue that is cited by response text in the previous round of dialogue, and the quality of the second cited segment does not meet requirements.

9. The apparatus of claim 8, wherein, When the quality type of the second cited segment is incorrect, the first retrieval knowledge is weakly related to the second cited segment. When the quality type of the second cited segment is a lack of content richness, the first retrieval knowledge is strongly related to the second cited segment and has higher content richness than the second cited segment.

10. The apparatus of any of claims 7-9, wherein, The processing module further performs the following operations: When the quality of the first cited segment meets requirements, the processing module performs one or more of the following operations: stores the first retrieval knowledge; generates a knowledge summary of the first session; or ends the first session.

11. The apparatus of any of claims 7-10, wherein, When the processing module performs reflection evaluation on the first retrieval knowledge based on the first retrieval knowledge, the first response text, and the first feedback of the user on the first response text to obtain the first reflection evaluation conclusion, the processing module specifically performs the following operations: calculates a co-occurrence segment between the first retrieval knowledge and the first response text to obtain the first cited segment; performs processing on the first cited segment and the first feedback by using a reflection evaluation model to obtain the first reflection evaluation conclusion.

12. The apparatus of any of claims 7-11, wherein, The first feedback is a follow-up question for the user in the next round of conversation.

13. A cluster of computing devices, characterized in that, comprise at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method of any of claims 1-6.

14. A computer-readable storage medium, characterized in that, comprise computer program instructions which, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method of any of claims 1-6, wherein the cluster of computing devices comprises at least one computing device.

15. A computer program product comprising instructions, characterized in that, comprise computer program instructions which, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method of any of claims 1-6, wherein the cluster of computing devices comprises at least one computing device.

Images