Multi-strategy intelligent question and answer method, system and equipment and medium

Abstract

The invention provides a multi-strategy intelligent question and answer method, system and device and a medium. The method comprises the steps that knowledge documents and real questions are obtained and processed into different document types to construct a knowledge base; filtering user questions according to desensitization detection, and returning a first target answer meeting a first threshold value based on a first recall strategy; if the first target answer does not meet the first threshold value, updating the user question according to the first target answer, executing a second recall strategy, and returning a second target answer meeting a second threshold value; and if the second target answer does not meet the second threshold value, rewriting the user question according to the knowledge base, executing a third recall strategy, and returning a third target answer of the user question.

Description

Technical Field

[0001] The present invention relates to the field of intelligent question answering technology, and in particular to a multi-strategy intelligent question answering method, system, device and medium. Background Art

[0002] The existing intelligent question-answering systems are mainly FAQ-based intelligent question-answering systems and RAG-based intelligent question-answering systems. The FAQ-based intelligent question-answering system summarizes and summarizes common questions and answers through data such as historical customer consultation records, product manuals and official documents, and then expands similar semantics based on the summarized questions to build an FAQ knowledge base, and uses query algorithms to find answers in the FAQ knowledge base. The RAG-based intelligent question-answering system parses, segments, cleans and stores the original documents to build a knowledge base, and rewrites user questions and inputs them into the LLM model to obtain answers.

[0003] However, the above-mentioned existing intelligent question systems still have some defects. First, the maintenance of the knowledge base in the FAQ-based intelligent question system is particularly time-consuming and resource-consuming, and the answers obtained based on the knowledge base are relatively limited; secondly, the system design in the RAG-based intelligent question-answering system is relatively complex, and each question needs to be rewritten, recalled, and generated, which consumes a lot of computing resources, and the overall response time is long and model hallucination problems are prone to occur.

[0004] Therefore, there is an urgent need for a multi-strategy intelligent question-answering method to solve the problem of insufficient question-answering performance caused by defects in existing intelligent question-answering systems. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention proposes a multi-strategy intelligent question-answering method, system, device and medium.

[0006] The first aspect of the present invention discloses a multi-strategy intelligent question answering method, comprising:

[0007] Acquire knowledge documents and real problems and process them into different document types to build a knowledge base;

[0008] Filter the user questions according to the desensitization detection, and return a first target answer that meets a first threshold based on a first recall strategy;

[0009] If the first target answer does not meet the first threshold, the user question is updated according to the first target answer to execute the second recall strategy, and the second target answer that meets the second threshold is returned;

[0010] If the second target answer does not meet the second threshold, the user question is rewritten according to the knowledge base to execute the third recall strategy and return the third target answer to the user question.

[0011] In an optional embodiment, the acquisition of knowledge documents and real questions is processed into different document types to construct a knowledge base, including:

[0012] After parsing the acquired knowledge documents, they are divided into blocks, the semantic information of each knowledge block is encoded to generate a knowledge vector, and the knowledge block and knowledge vector are combined and recorded as a doc document type and saved in the knowledge base;

[0013] Input the doc document in the knowledge base into the LLM model to generate a first QA pair including a question and an answer, and after confirming the QA pair, record it as a qa document type and save it in the knowledge base;

[0014] The real question is rewritten through the RAG mode and the real answer of the real question is generated. After the rewritten question and the real answer are confirmed, they are set as the second QA pair and saved in the knowledge base, and the document type of the second QA pair is recorded as qa type.

[0015] In an optional embodiment, filtering user questions according to desensitization detection includes:

[0016] Set matching keywords for sensitive words in user questions to determine whether to filter, and return safe answers to user questions that need to be filtered;

[0017] For identity recognition and chat type recognition in user questions, the knowledge base is used to calculate the distance of the semantic information of the user questions in the vector space, and the recognition threshold is set to determine whether to filter, and safe answers are returned to user questions that need to be filtered.

[0018] In an optional embodiment, returning a first target answer that meets a first threshold based on a first recall strategy includes:

[0019] According to the user's question, a recall algorithm is used in the qa-type documents of the knowledge base to obtain a recall segment matching the user's question in the QA pair, wherein the recall segment includes a question, an answer, and a question and an answer;

[0020] The KMS threshold is fine-tuned according to the recall accuracy and recall rate to be configured as the first threshold under the first recall strategy, and the recall fragments exceeding the first threshold are selected as the first target answer of the first recall strategy.

[0021] In an optional embodiment, if the first target answer does not meet the first threshold, updating the user question according to the first target answer to execute the second recall strategy and returning the second target answer that meets the second threshold includes:

[0022] The topk recalled segments of the first target answer are obtained and set as the first context. The first context is concatenated with the user question and then input into the LLM model to generate the second target answer.

[0023] The F1 score and the prompt word are set as the second threshold for the second target answer, and the validity detection and hallucination detection are performed on the second target answer through the second threshold.

[0024] In an optional embodiment, if the second target answer does not meet the second threshold, rewriting the user question according to the knowledge base to execute a third recall strategy and returning a third target answer to the user question includes:

[0025] The user questions are rewritten into k new query questions according to the step-back strategy, hyde strategy and description conversion strategy. The qa documents and doc documents in the knowledge base are used to recall the query questions. The recalled contents are precisely sorted by the ranking fusion method to obtain the topk recalled contents. The topk recalled contents after precise sorting are spliced ​​with the user questions as context and then input into the LLM model to generate the third target answer.

[0026] The F1 score and the prompt word are set as the third threshold for the third target answer, and the validity detection and hallucination detection are performed on the third target answer through the third threshold.

[0027] In an optional embodiment, if the second target answer does not meet the second threshold, rewriting the user question according to the knowledge base to execute a third recall strategy and returning the third target answer to the user question also includes:

[0028] If the third target answer does not meet the third threshold, the LLM model is adjusted to follow the restrictions on the topk recalled content contexts, and the user question is re-input into the LLM model to generate the fourth target answer.

[0029] The second aspect of the present invention discloses a multi-strategy intelligent question-answering system, the system comprising:

[0030] The knowledge base module is used to obtain knowledge documents and process real problems into different document types to build a knowledge base;

[0031] A first strategy module, configured to filter user questions according to desensitization detection, and return a first target answer that meets a first threshold based on a first recall strategy;

[0032] A second strategy module is used for updating the user question according to the first target answer to execute a second recall strategy and return a second target answer that meets the second threshold if the first target answer does not meet the first threshold;

[0033] The third strategy module is used to rewrite the user question according to the knowledge base to execute a third recall strategy and return a third target answer to the user question if the second target answer does not meet the second threshold.

[0034] In an optional embodiment, the knowledge base module includes:

[0035] The knowledge base submodule is used to parse the acquired knowledge documents and divide them into blocks, encode the semantic information of each knowledge block to generate a knowledge vector, record the knowledge block and knowledge vector combination as a doc document type and save it in the knowledge base; input the doc document in the knowledge base into the LLM model to generate a first QA pair including a question and an answer, and after confirming the QA pair, record it as a qa document type and save it in the knowledge base; rewrite the question content of the real question through the RAG mode and generate the real answer to the real question, and after confirming the rewritten question and the real answer, set it as a second QA pair and save it in the knowledge base, and record the document type of the second QA pair as qa type.

[0036] In an optional embodiment, the first policy module includes:

[0037] The desensitization module is used to set matching keywords for sensitive words in user questions to determine whether to filter, and return safe answers to user questions that need to be filtered; for identity recognition and chat type recognition in user questions, the knowledge base is used to calculate the distance of the semantic information of the user questions in the vector space, and the recognition threshold is set to determine whether to filter, and return safe answers to user questions that need to be filtered.

[0038] In an optional embodiment, the first policy module includes:

[0039] The first strategy submodule is used to obtain a recall segment matching the user question in the QA pair by using a recall algorithm in the qa-type documents of the knowledge base according to the user question, wherein the recall segment includes a question, an answer, and a question and an answer;

[0040] The KMS threshold is fine-tuned according to the recall accuracy and recall rate to be configured as the first threshold under the first recall strategy, and the recall fragments exceeding the first threshold are selected as the first target answer of the first recall strategy.

[0041] In an optional embodiment, the second policy module includes:

[0042] The second strategy submodule is used to obtain the topk recalled fragments of the first target answer and set them as the first context. The first context is concatenated with the user question and then input into the LLM model to generate the second target answer. The F1 score and the prompt word are set as the second threshold for the second target answer, and the validity detection and hallucination detection are performed on the second target answer through the second threshold.

[0043] In an optional embodiment, the third policy module includes:

[0044] The third strategy submodule is used to rewrite user questions into k new query questions according to the step-back strategy, hyde strategy and description conversion strategy, use the qa documents and doc documents in the knowledge base to recall the query questions, and accurately sort the recalled contents by ranking fusion method to obtain the topk recalled contents, and then input the accurately sorted topk recalled contents into the LLM model after splicing with the user questions as the context to generate the third target answer; set the F1 score and prompt word as the third threshold for the third target answer, and perform validity detection and hallucination detection on the third target answer through the third threshold.

[0045] In an optional embodiment, the third policy module includes:

[0046] The fourth strategy submodule is used to adjust the LLM model's compliance restrictions on the topk recalled content contexts if the third target answer does not meet the third threshold, and re-input the user question into the LLM model to generate the fourth target answer.

[0047] The third aspect of the present invention discloses a multi-strategy intelligent question-answering device, comprising:

[0048] at least one processor, and

[0049] a memory communicatively connected to the at least one processor; wherein,

[0050] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multi-strategy intelligent question-answering method as described in any one of the first aspects of the present invention.

[0051] The fourth aspect of the present invention discloses a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the multi-strategy intelligent question-answering method as described in any one of the first aspect of the present invention.

[0052] Compared with the prior art, the present invention has the following advantages:

[0053] The present invention proposes a multi-strategy intelligent question-answering method, system, device and medium, the method comprising: obtaining knowledge documents and real questions and processing them into different document types to construct a knowledge base; filtering user questions according to desensitization detection, and returning a first target answer that meets a first threshold based on a first recall strategy; if the first target answer does not meet the first threshold, updating the user question according to the first target answer to execute a second recall strategy, and returning a second target answer that meets a second threshold; if the second target answer does not meet the second threshold, rewriting the user question according to the knowledge base to execute a third recall strategy, and returning a third target answer to the user question. Combining the advantages of FAQ mode and RAG mode, a high-quality knowledge base is constructed by generating documents, and different recall strategies are dynamically used for questions of different difficulty levels. In particular, when the knowledge base cannot answer the question, the model can be actively asked to use other knowledge to answer, which greatly improves the recall effect while ensuring the performance of the question-answering system, and at the same time reduces the total response time of the system, improves the comprehensive response efficiency, and has a high application value in the field of intelligent question-answering. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0055] Figure 1 It is a flow chart of the multi-strategy intelligent question-answering method of the present invention;

[0056] Figure 2 Schematic diagram of the multi-strategy intelligent question answering system of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0058] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0059] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0060] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0061] Example 1

[0062] See also Figure 1 The embodiment of the present invention discloses a multi-strategy intelligent question answering method, including:

[0063] 101. Obtain knowledge documents and process real problems into different document types to build a knowledge base;

[0064] It should be noted that knowledge documents usually include but are not limited to historical customer consultation records, product manuals, official documents and other files with knowledge attributes, which are mainly used to build a knowledge base that references the knowledge required to answer questions, and real questions often include questions related to professional knowledge to expand the knowledge reserve of the knowledge base. In order to further improve the upper limit and robustness of the standard answer of the knowledge base, knowledge is stored in different document types to facilitate the subsequent return of the standard answer corresponding to the input question by calculating the similarity. The intelligent question-answering system in this application mainly solves professional knowledge problems, such as knowledge problems in the CRM field, and the corresponding collected knowledge documents and real problems will be processed around the CRM field.

[0065] In an optional embodiment, the acquisition of knowledge documents and real questions is processed into different document types to construct a knowledge base, including:

[0066] After parsing the acquired knowledge documents, they are divided into blocks, the semantic information of each knowledge block is encoded to generate a knowledge vector, and the knowledge block and knowledge vector are combined and recorded as a doc document type and saved in the knowledge base;

[0067] Input the doc document in the knowledge base into the LLM model to generate a first QA pair including a question and an answer, and after confirming the QA pair, record it as a qa document type and save it in the knowledge base;

[0068] The real question is rewritten through the RAG mode and the real answer of the real question is generated. After the rewritten question and the real answer are confirmed, they are set as the second QA pair and saved in the knowledge base, and the document type of the second QA pair is recorded as qa type.

[0069] Specifically, an embedding model can be used to encode each knowledge block, that is, by representing each data point in the knowledge block as a vector, the voice information of the data is encoded into the vector space, and the correlation and similarity between the data are captured by the vector, so that similar data are closer in the vector space. The knowledge documents after embedding processing are saved in the doc document type to distinguish them from other data in the knowledge base, so as to facilitate the selection of appropriate data when recalling user questions later.

[0070] Furthermore, in addition to using the doc documents generated by the knowledge documents, the doc documents are also input into the LLM (Large Language Model) large language model, and the doc documents are used as training data. The LLM model generates more relevant questions and related answers that can be used as knowledge references. The generalization ability of the knowledge base is improved through the LLM model, and the robustness of intelligent question and answer is enhanced. The corresponding questions and answers are saved as qa type documents in the form of QA, which is different from the doc documents and facilitates the subsequent recall process to call the data in the knowledge base.

[0071] Furthermore, real questions are frequently asked questions in professional fields obtained from historical data, reports or surveys that are not generated by the LLM model. The processing method of real questions refers to the intelligent question and answer system under the RAG mode to rewrite the real questions into several basic questions or simple questions. For each simple question and basic question, the topk fragments are recalled in the knowledge base as the context of the real question, and the context is spliced ​​with the real question and input into the LLM model to obtain the real answer. Since the real answer is obtained based on the real question, the rewritten question and the real answer are entered into the knowledge base as a QA pair and recorded as a qa type document to distinguish other documents. Through the expansion of the above-mentioned multiple knowledge bases, the loss of building and maintaining the knowledge base can be reduced as much as possible, and the range of answers covered by the knowledge base can be increased, thereby enhancing the robustness of the intelligent question and answer system. For simple questions in professional knowledge, compared with consuming a large amount of computing resources through the steps of the RAG mode, some basic questions can be answered through the knowledge base first.

[0072] Furthermore, the above-mentioned various types of documents can be classified and managed through the elastic search component Elasticsearch. Specifically, Elasticsearch can pre-process questions, mainly including word segmentation and part-of-speech tagging, and Elasticsearch can also provide rich query DSL (domain-specific language), which can achieve accurate matching of keywords in questions and in-depth understanding of semantics by building complex query statements. This helps the question-answering system to accurately understand the user's intentions and needs, so as to give more accurate answers. For different document types such as doc documents and qa documents, Elasticsearch adds tags to questions according to the content and business scenarios of the questions, so as to facilitate the classification management of question-answer pairs. Elasticsearch supports custom fields and index types, which can easily realize tag classification and management. By adding tags to questions, similar questions can be matched under the tag classification, so as to improve the machine's understanding ability and matching accuracy, so that the question-answering system can process large-scale data and respond to user needs quickly.

[0073] 102. Filter the user question according to the desensitization detection, and return a first target answer that meets a first threshold based on a first recall strategy;

[0074] It should be noted that the intelligent question and answer system mainly solves professional knowledge problems. For example, knowledge problems in the CRM field, the corresponding collected knowledge documents and real problems will be processed around the CRM field. In order to avoid unnecessary resource loss, when receiving user input questions, desensitization detection is required first. The specific detection content includes but is not limited to sensitive word detection, identity recognition detection and chat recognition detection, which is mainly used to distinguish whether the user's purpose is to query the answer corresponding to the professional knowledge question. For filtered user questions, simple questions are filtered through the first recall strategy, that is, partial answers can be returned, and it can also be further judged whether the strategy needs to be changed to distinguish the complexity of user questions, thereby reducing computing loss and improving response efficiency.

[0075] In an optional embodiment, filtering user questions according to desensitization detection includes:

[0076] Set matching keywords for sensitive words in user questions to determine whether to filter, and return safe answers to user questions that need to be filtered;

[0077] For identity recognition and chat type recognition in user questions, the knowledge base is used to calculate the distance of the semantic information of the user questions in the vector space, and the recognition threshold is set to determine whether to filter, and safe answers are returned to user questions that need to be filtered.

[0078] Specifically, for sensitive words, keywords involving sensitive words can be set for filtering, while for identity recognition and small talk, the purpose of the user's question can be determined through the QA pairs in the knowledge base. If it is identity recognition (for example, asking: Is the question-and-answer system an OPENAI system?), such questions will not be answered in order to protect privacy. If it is a small talk type of user question, only questions related to professional knowledge will be answered in order to reduce the consumption of computing resources. In the above situations, safe answers will be returned, for example (I am a robot with CRM professional knowledge and cannot answer your question. Please ask other questions related to CRM).

[0079] In an optional embodiment, returning a first target answer that meets a first threshold based on a first recall strategy includes:

[0080] According to the user's question, a recall algorithm is used in the qa-type documents of the knowledge base to obtain a recall segment matching the user's question in the QA pair, wherein the recall segment includes a question, an answer, and a question and an answer;

[0081] The KMS threshold is fine-tuned according to the recall accuracy and recall rate to be configured as the first threshold under the first recall strategy, and the recall fragments exceeding the first threshold are selected as the first target answer of the first recall strategy.

[0082] Specifically, for the user question received for the first time, the corresponding answer is recalled by the knowledge base for a preliminary answer, or it can be understood as a simple answer. In addition to setting the first threshold through the KMS threshold, the recalled content can also be screened by setting rules to select the standard answer that meets the user question. Since the knowledge base distinguishes different document types when setting, compared with other types of knowledge documents, qa type documents can establish connections with user questions more quickly, thereby quickly responding to and returning the standard answer to the user question. In this way, there is no need to judge whether the strategy meets the requirements of the user question. Instead, the corresponding answer is generated first and then the rationality of the answer is examined, which reduces unnecessary computing loss and improves the performance of intelligent question and answer. If the returned answer cannot meet the user question, it means that the user question is not a simple question, and other deepening strategies are needed to return the answer to the complex question.

[0083] 103. If the first target answer does not meet the first threshold, update the user question according to the first target answer, execute the second recall strategy, and return the second target answer that meets the second threshold;

[0084] In an optional embodiment, if the first target answer does not meet the first threshold, updating the user question according to the first target answer to execute the second recall strategy and returning the second target answer that meets the second threshold includes:

[0085] The topk recalled segments of the first target answer are obtained and set as the first context. The first context is concatenated with the user question and then input into the LLM model to generate the second target answer.

[0086] The F1 score and the prompt word are set as the second threshold for the second target answer, and the validity detection and hallucination detection are performed on the second target answer through the second threshold.

[0087] Specifically, if the first strategy cannot generate an answer to the user's question, it is possible that the user's question is not a simple question but may be composed of but not limited to multiple simple questions or the question lacks context. Therefore, when the first target answer is not met, the topk recalled fragments returned by the first strategy can be used as the knowledge basis to supplement the user's question context or the knowledge basis of multiple simple questions. The topk recalled fragments are inserted into the user's question as context, where context is a crucial concept in the question-answering system. It can help the question-answering system understand the user's intention more accurately when the user queries, thereby providing a more prepared answer. In this embodiment, the description of the user's question is further enriched through the context to facilitate the LLM model to understand the user's intention. The LLM model itself can obtain the most relevant text content of the user's question through training corpus and external knowledge base for learning, and finally combine the relevant content with the question to generate the final answer. Therefore, it can solve difficult problems that the first recall strategy cannot answer. The second recall strategy uses the recall fragments generated in the first recall strategy, which can make the question closer to the user's true intention. This process also reduces the resource loss of the question-answering system calculation and improves system performance.

[0088] Furthermore, the second threshold is mainly used to detect the feasibility of the result of the second target answer, and whether the answer generated by the LLM model has the problem of model hallucination. The F1 score mainly includes the correctness of the answer generated by the model (evaluating the true correctness of the response content generated by the model), completeness (measuring the completeness of the type of output information that the model can provide based on the user's input information), relevance (characterizing the relevance between the output information of the model based on a series of related input information from the user) and effectiveness (evaluating the usefulness of the response content generated by the model for the user's input information). Specific scores such as correctness score are achieved by comparing with known facts or standard answers, while correctness score is achieved by comparing with known facts or standard answers. Indicators are usually divided into multiple levels, such as a 6-level scoring system with a full score of 5 points, and the harmonic mean of precision and recall to estimate the accuracy of the model in tasks such as classification, extraction, and error correction. For the hallucination problem of the LLM model, prompt words can be used, such as asking the large language model whether the answer is generated from an incomplete or ambiguous context, or a static prompt test can be used to create a set of prompts related to the application scenario and benchmark the model's response. By manually evaluating the correctness and coherence of the output, it is possible to identify whether the model has hallucination problems. The combination of the above tests can improve the accuracy and reliability of the model's output answers and reduce the impact of model hallucinations on generated answers.

[0089] 104. If the second target answer does not meet the second threshold, rewrite the user question according to the knowledge base to execute the third recall strategy and return the third target answer to the user question.

[0090] It should be noted that if the second target answer cannot meet the second threshold, there may be some problems. For example, in the second recall strategy, the large language model will pay too much attention to a basic question or a simple question of the user's question after splicing the context, resulting in the second target answer being unsatisfactory. Therefore, it is necessary to continue to adjust the recall strategy.

[0091] In an optional embodiment, if the second target answer does not meet the second threshold, rewriting the user question according to the knowledge base to execute a third recall strategy and returning a third target answer to the user question includes:

[0092] The user questions are rewritten into k new query questions according to the step-back strategy, hyde strategy and description conversion strategy. The qa documents and doc documents in the knowledge base are used to recall the query questions. The recalled contents are precisely sorted by the ranking fusion method to obtain the topk recalled contents. The topk recalled contents after precise sorting are spliced ​​with the user questions as context and then input into the LLM model to generate the third target answer.

[0093] The F1 score and the prompt word are set as the third threshold for the third target answer, and the validity detection and hallucination detection are performed on the third target answer through the third threshold.

[0094] Specifically, in order to make the LLM large language model further understand user questions and reduce the attention of the large language model to a certain question, step-back can be used to degenerate user questions into simplified questions that are easier for the large language model to understand in order to improve the answering effect. For example, before asking user questions, a more basic or intuitive question will be asked to help the LLM large language model view and understand the original question of the user question from different perspectives. Therefore, the question originally input by the user is rewritten into k query questions after step-back processing, and the query is used as input in multiple document types in the knowledge base (such as qa type documents and doc type documents) for recall at the same time, and the recalled content of the query question after step-back is obtained as support as much as possible. The hypothetical document using the hyde (Hypothetical Document Embeddings) strategy is regarded as the "hypothetical answer" of the query, so as to generate more rich contexts with hypothetical attributes, thereby expanding the quantity and quality of query questions. The description conversion strategy is used to transform the words used to generate query questions, and generate different description methods for query questions with the same meaning to expand the data of query questions, which is convenient for the model to understand. RRF (Reciprocal The ranking fusion method of RRF will merge and sort the recall content results obtained according to different document type queries. Since the RRF ranking fusion method can take into account the position of the queried recall content in the original ranking (i.e., the ranking of the topk recall content returned by different document types), ranking fusion will give higher-ranked recall content in different result lists higher importance, thereby effectively integrating the results of multiple retrievers to improve the overall performance and accuracy of the question-answering system. Since the second recall strategy cannot obtain a satisfactory answer, in order to improve the effectiveness and accuracy of the output answer, the computational loss of the system can be appropriately sacrificed. However, in this method, multiple strategies are used to hierarchically handle user problems. Therefore, unless it is necessary to use the third recall strategy, other recall strategies can solve most user problems, which reduces the computational loss of the question-answering system as a whole. In the actual process, the output of the previous level strategy also provides reference knowledge for the input of the next strategy, avoiding the resource loss caused by repeated calculations. Finally, the topk recall contents after fusion ranking are spliced ​​with the user question and input into the LLM large language model to generate the third target answer (i.e., the answer to the difficult question).

[0095] Furthermore, in the above steps, the large language model and its output results also need to be tested for validity and hallucination. It can be understood that except for further adjusting the weight of the evaluation score and the rules of the prompt words, the detection process of the third recall strategy is roughly the same as the detection process of the second recall strategy.

[0096] In an optional embodiment, if the second target answer does not meet the second threshold, rewriting the user question according to the knowledge base to execute a third recall strategy and returning the third target answer to the user question also includes:

[0097] If the third target answer does not meet the third threshold, the LLM model is adjusted to follow the restrictions on the topk recalled content contexts, and the user question is re-input into the LLM model to generate the fourth target answer.

[0098] Specifically, in order to limit the large language model from outputting answers that are divorced from the user's questions, some rules that need to be followed are usually pre-set before inputting the user's questions. If the third recall strategy still cannot generate a satisfactory answer to the user's question, the compliance with the recalled content can be relaxed, which can be understood as increasing the range of optional recalled content, thereby providing users with satisfactory answers to questions from more optional answers. It can be understood that in the above steps, except for adjusting the model compliance restrictions, the other steps are consistent with the third recall strategy, but since the compliance restrictions are relaxed, the actually generated answers will be richer, thereby outputting satisfactory answers to user questions as much as possible.

[0099] The present invention proposes a multi-strategy intelligent question-answering method, system, device and medium, the method comprising: obtaining knowledge documents and real questions and processing them into different document types to construct a knowledge base; filtering user questions according to desensitization detection, and returning a first target answer that meets a first threshold based on a first recall strategy; if the first target answer does not meet the first threshold, updating the user question according to the first target answer to execute a second recall strategy, and returning a second target answer that meets a second threshold; if the second target answer does not meet the second threshold, rewriting the user question according to the knowledge base to execute a third recall strategy, and returning a third target answer to the user question. Combining the advantages of FAQ mode and RAG mode, a high-quality knowledge base is constructed by generating documents, and different recall strategies are dynamically used for questions of different difficulty levels. In particular, when the knowledge base cannot answer the question, the model can be actively asked to use other knowledge to answer, which greatly improves the recall effect while ensuring the performance of the question-answering system, and at the same time reduces the total response time of the system, improves the comprehensive response efficiency, and has a high application value in the field of intelligent question-answering.

[0100] like Figure 2 As shown, the second aspect of the present invention discloses a multi-strategy intelligent question answering system, the system comprising:

[0101] The knowledge base module is used to obtain knowledge documents and process real problems into different document types to build a knowledge base;

[0102] A first strategy module, configured to filter user questions according to desensitization detection, and return a first target answer that meets a first threshold based on a first recall strategy;

[0103] A second strategy module is used for updating the user question according to the first target answer to execute a second recall strategy and return a second target answer that meets the second threshold if the first target answer does not meet the first threshold;

[0104] The third strategy module is used to rewrite the user question according to the knowledge base to execute a third recall strategy and return a third target answer to the user question if the second target answer does not meet the second threshold.

[0105] In an optional embodiment, the knowledge base module includes:

[0106] The knowledge base submodule is used to parse the acquired knowledge documents and divide them into blocks, encode the semantic information of each knowledge block to generate a knowledge vector, record the knowledge block and knowledge vector combination as a doc document type and save it in the knowledge base; input the doc document in the knowledge base into the LLM model to generate a first QA pair including a question and an answer, and after confirming the QA pair, record it as a qa document type and save it in the knowledge base; rewrite the question content of the real question through the RAG mode and generate the real answer to the real question, and after confirming the rewritten question and the real answer, set it as a second QA pair and save it in the knowledge base, and record the document type of the second QA pair as qa type.

[0107] In an optional embodiment, the first policy module includes:

[0108] The desensitization module is used to set matching keywords for sensitive words in user questions to determine whether to filter, and return safe answers to user questions that need to be filtered; for identity recognition and chat type recognition in user questions, the knowledge base is used to calculate the distance of the semantic information of the user questions in the vector space, and the recognition threshold is set to determine whether to filter, and return safe answers to user questions that need to be filtered.

[0109] In an optional embodiment, the first policy module includes:

[0110] The first strategy submodule is used to obtain a recall segment matching the user question in the QA pair by using a recall algorithm in the qa-type documents of the knowledge base according to the user question, wherein the recall segment includes a question, an answer, and a question and an answer;

[0111] The KMS threshold is fine-tuned according to the recall accuracy and recall rate to be configured as the first threshold under the first recall strategy, and the recall fragments exceeding the first threshold are selected as the first target answer of the first recall strategy.

[0112] In an optional embodiment, the second policy module includes:

[0113] The second strategy submodule is used to obtain the topk recalled fragments of the first target answer and set them as the first context. The first context is concatenated with the user question and then input into the LLM model to generate the second target answer. The F1 score and the prompt word are set as the second threshold for the second target answer, and the validity detection and hallucination detection are performed on the second target answer through the second threshold.

[0114] In an optional embodiment, the third policy module includes:

[0115] The third strategy submodule is used to rewrite user questions into k new query questions according to the step-back strategy, hyde strategy and description conversion strategy, use the qa documents and doc documents in the knowledge base to recall the query questions, and accurately sort the recalled contents by ranking fusion method to obtain the topk recalled contents, and then input the accurately sorted topk recalled contents into the LLM model after splicing with the user questions as the context to generate the third target answer; set the F1 score and prompt word as the third threshold for the third target answer, and perform validity detection and hallucination detection on the third target answer through the third threshold.

[0116] In an optional embodiment, the third policy module includes:

[0117] The fourth strategy submodule is used to adjust the LLM model's compliance restrictions on the topk recalled content contexts if the third target answer does not meet the third threshold, and re-input the user question into the LLM model to generate the fourth target answer.

[0118] The third aspect of the present invention discloses a multi-strategy intelligent question-answering device, comprising:

[0119] at least one processor, and

[0120] a memory communicatively connected to the at least one processor; wherein,

[0121] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multi-strategy intelligent question-answering method as described in any one of the first aspects of the present invention.

[0122] The computer device may be a terminal, and the computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multi-strategy intelligent question-answering method is implemented. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0123] The fourth aspect of the present invention discloses a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the multi-strategy intelligent question-answering method as described in any one of the first aspect of the present invention.

[0124] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it may include the processes of the embodiments of the above-mentioned intelligent question-answering method through multiple strategies. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0125] Alternatively, if the above modules of the present invention are implemented in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, RAM, ROM, magnetic disks or optical disks.

[0126] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-strategy intelligent question answering method, characterized in that: The method comprises: Acquire knowledge documents and real problems and process them into different document types to build a knowledge base; Filter the user questions according to the desensitization detection, and return a first target answer that meets a first threshold based on a first recall strategy; If the first target answer does not meet the first threshold, the user question is updated according to the first target answer to execute the second recall strategy, and the second target answer that meets the second threshold is returned; If the second target answer does not meet the second threshold, the user question is rewritten according to the knowledge base to execute the third recall strategy and return the third target answer to the user question.

2. The multi-strategy intelligent question-answering method according to claim 1, characterized in that: The acquired knowledge documents and real questions are processed into different document types to build a knowledge base, including: After parsing the acquired knowledge documents, they are divided into blocks, the semantic information of each knowledge block is encoded to generate a knowledge vector, and the knowledge block and knowledge vector are combined and recorded as a doc document type and saved in the knowledge base; Input the doc document in the knowledge base into the LLM model to generate a first QA pair including a question and an answer, and after confirming the QA pair, record it as a qa document type and save it in the knowledge base; The real question is rewritten through the RAG mode and the real answer of the real question is generated. After the rewritten question and the real answer are confirmed, they are set as the second QA pair and saved in the knowledge base, and the document type of the second QA pair is recorded as qa type.

3. The multi-strategy intelligent question-answering method according to claim 1, characterized in that: The filtering of user questions based on desensitization detection includes: Set matching keywords for sensitive words in user questions to determine whether to filter, and return safe answers to user questions that need to be filtered; For identity recognition and chat type recognition in user questions, the knowledge base is used to calculate the distance of the semantic information of the user questions in the vector space, and the recognition threshold is set to determine whether to filter, and a safe answer is returned to the user questions that need to be filtered.

4. The multi-strategy intelligent question-answering method according to claim 1, characterized in that: The step of returning a first target answer that meets a first threshold based on a first recall strategy includes: According to the user's question, a recall algorithm is used in the qa-type documents of the knowledge base to obtain a recall segment matching the user's question in the QA pair, wherein the recall segment includes a question, an answer, and a question and an answer; The KMS threshold is fine-tuned according to the recall accuracy and recall rate to be configured as the first threshold under the first recall strategy, and the recall fragments exceeding the first threshold are selected as the first target answer of the first recall strategy.

5. The multi-strategy intelligent question-answering method according to claim 1, characterized in that: If the first target answer does not meet the first threshold, updating the user question according to the first target answer to execute the second recall strategy and returning the second target answer that meets the second threshold, including: The topk recalled segments of the first target answer are obtained and set as the first context. The first context is concatenated with the user question and then input into the LLM model to generate the second target answer. The F1 score and the prompt word are set as the second threshold for the second target answer, and the validity detection and hallucination detection are performed on the second target answer through the second threshold.

6. The multi-strategy intelligent question-answering method according to claim 1, characterized in that: If the second target answer does not meet the second threshold, rewriting the user question according to the knowledge base to execute the third recall strategy and returning the third target answer to the user question includes: The user questions are rewritten into k new query questions according to the step-back strategy, hyde strategy and description conversion strategy. The qa documents and doc documents in the knowledge base are used to recall the query questions. The recalled contents are precisely sorted by the ranking fusion method to obtain the topk recalled contents. The topk recalled contents after precise sorting are spliced ​​with the user questions as context and then input into the LLM model to generate the third target answer. The F1 score and the prompt word are set as the third threshold for the third target answer, and the validity detection and hallucination detection are performed on the third target answer through the third threshold.

7. The multi-strategy intelligent question-answering method according to claim 1, characterized in that: If the second target answer does not meet the second threshold, rewriting the user question according to the knowledge base to execute the third recall strategy and returning the third target answer to the user question also includes: If the third target answer does not meet the third threshold, the LLM model is adjusted to follow the restrictions on the topk recalled content contexts, and the user question is re-input into the LLM model to generate the fourth target answer.

8. A multi-strategy intelligent question-answering system, characterized in that: The system comprises: The knowledge base module is used to obtain knowledge documents and process real problems into different document types to build a knowledge base; A first strategy module, configured to filter user questions according to desensitization detection, and return a first target answer that meets a first threshold based on a first recall strategy; A second strategy module is used for updating the user question according to the first target answer to execute a second recall strategy and return a second target answer that meets the second threshold if the first target answer does not meet the first threshold; The third strategy module is used to rewrite the user question according to the knowledge base to execute a third recall strategy and return a third target answer to the user question if the second target answer does not meet the second threshold.

9. A multi-strategy intelligent question-answering device, characterized in that: include: at least one processor, and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multi-strategy intelligent question-answering method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the multi-strategy intelligent question-answering method as described in any one of claims 1 to 7.

Images