A multi-strategy intelligent question-answering method, system, device and medium

Abstract

This invention proposes a multi-strategy intelligent question answering method, system, device, and medium. The method includes: acquiring knowledge documents and processing real questions into different document types to construct a knowledge base; filtering user questions based on de-identification detection, and returning a first target answer that meets a first threshold based on a first recall strategy; if the first target answer does not meet the first threshold, updating the user question based on the first target answer and executing a second recall strategy to return a second target answer that meets the second threshold; if the second target answer does not meet the second threshold, rewriting the user question based on the knowledge base and executing a third recall strategy to return a third target answer to the user question.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent question answering, and in particular to a multi-strategy intelligent question answering method, system, device and medium. BACKGROUND

[0002] The existing intelligent question answering system is mainly a FAQ-based intelligent question answering system and a RAG-based intelligent question answering system. The FAQ-based intelligent question answering system is to summarize and induce common questions and answers from historical customer consultation records, product manuals and official documents, etc., and then expand similar semantics according to the induced questions to build a FAQ knowledge base, and use a query algorithm to find answers in the FAQ knowledge base. The RAG-based intelligent question answering system builds a knowledge base after parsing, blocking, cleaning and warehousing of original documents, and inputs the rewritten user question into a LLM model to obtain an answer.

[0003] However, the existing intelligent question answering system still has some defects. First, the maintenance of the knowledge base in the FAQ-based intelligent question answering system consumes a lot of time and resources, and the answers obtained based on the knowledge base are limited. Second, the system design of the RAG-based intelligent question answering system is complex, and it needs to rewrite, recall and generate for each question, which consumes a lot of computing resources and has a long overall response time and is prone to model hallucination problems.

[0004] Therefore, there is an urgent need for a multi-strategy intelligent question answering method to solve the problem of insufficient question answering performance caused by the defects of the existing intelligent question answering system. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides a multi-strategy intelligent question answering method, system, device and medium.

[0006] The first aspect of the present application discloses a multi-strategy intelligent question answering method, comprising:

[0007] Processing the knowledge documents and the real questions into different document types to build a knowledge base;

[0008] Filtering the user question according to the desensitization detection, and returning a first target answer satisfying a first threshold value based on a first recall strategy;

[0009] If the first target answer does not satisfy the first threshold value, updating the user question according to the first target answer to execute a second recall strategy, and returning a second target answer satisfying a second threshold value;

[0010] If the second target answer does not satisfy the second threshold value, rewriting the user question according to the knowledge base to execute a third recall strategy, and returning a third target answer of the user question.

[0011] In an optional embodiment, the acquired knowledge documents and real questions are processed as different document types to build a knowledge base, including:

[0012] After parsing the acquired knowledge documents, the knowledge blocks are processed in blocks, the semantic information of each knowledge block is encoded to generate a knowledge vector, and the knowledge block and the knowledge vector are combined and recorded as a doc document type and saved to the knowledge base;

[0013] The doc document in the knowledge base is input into an LLM model to generate a first QA pair including a question and an answer, and the QA pair is confirmed and recorded as a qa document type and saved to the knowledge base;

[0014] The real question is rewritten by a RAG mode to generate a real answer to the real question, and the rewritten question and the real answer are confirmed and set as a second QA pair and saved to the knowledge base, and the document type of the second QA pair is recorded as a qa type.

[0015] In an optional embodiment, the user question is filtered according to desensitization detection, including:

[0016] Sensitive words in the user question are matched to determine whether to filter, and a safe answer is returned for the user question that needs to be filtered;

[0017] Identity recognition and casual type recognition in the user question are used to calculate the semantic information of the user question in the vector space by using the knowledge base, a recognition threshold is set to determine whether to filter, and a safe answer is returned for the user question that needs to be filtered.

[0018] In an optional embodiment, the first target answer satisfying a first threshold is returned based on a first recall strategy, including:

[0019] According to the user question, a recall fragment matching the user question in the QA pair is obtained in the qa document of the knowledge base by using a recall algorithm, and the recall fragment includes a question, an answer, and the question and the answer;

[0020] The KMS threshold is fine-tuned according to the recall accuracy and the recall rate to be the first threshold under the first recall strategy, and the recall fragment exceeding the first threshold is selected as the first target answer of the first recall strategy.

[0021] In an optional embodiment, if the first target answer does not satisfy the first threshold, a second recall strategy is executed according to the first target answer to return a second target answer satisfying a second threshold, including:

[0022] The topk recall fragments of the first target answer are set as a first context, the first context is spliced with the user question, and the LLM model is input to generate a second target answer;

[0023] The F1 score and the cue word are set as the second threshold for the second target answer. The validity and illusion detection of the second target answer are performed by using the second threshold.

[0024] In an optional embodiment, the step of rewriting the user question according to the knowledge base and executing a third recall strategy to return the third target answer to the user question if the second target answer does not meet the second threshold includes:

[0025] The user question is rewritten into k new query questions based on the step-back strategy, hyde strategy, and description transformation strategy. The query questions are recalled using QA and DOC documents in the knowledge base. The recalled content is ranked and fused to obtain the top k recalled content. The top k recalled content is concatenated with the user question as context and input into the LLM model to generate the third target answer.

[0026] The F1 score and the prompt word are set as the third threshold for the third target answer. The validity and illusion detection of the third target answer are performed by using the third threshold.

[0027] In an optional embodiment, the step of rewriting the user question according to the knowledge base and executing a third recall strategy to return the third target answer to the user question if the second target answer does not meet the second threshold further includes:

[0028] If the third target answer does not meet the third threshold, the LLM model's adherence to the context of the top k recalled content is adjusted, and the user question is re-entered into the LLM model to generate the fourth target answer.

[0029] A second aspect of this invention discloses a multi-strategy intelligent question-answering system, the system comprising:

[0030] The knowledge base module is used to acquire knowledge documents and real-world problem solutions, categorized into different document types to build the knowledge base.

[0031] The first strategy module is used to filter user questions based on de-identification detection and return the first target answer that meets the first threshold based on the first recall strategy.

[0032] The second strategy module is used to update the user question based on the first target answer and execute the second recall strategy if the first target answer does not meet the first threshold, and return the second target answer that meets the second threshold.

[0033] The third strategy module is used to rewrite the user question based on the knowledge base and execute the third recall strategy if the second target answer does not meet the second threshold, returning the third target answer to the user question.

[0034] In an optional embodiment, the knowledge base module includes:

[0035] The knowledge base submodule is used to parse and segment the acquired knowledge documents, encode the semantic information of each knowledge block to generate a knowledge vector, and combine the knowledge block and knowledge vector into a doc document type and save it to the knowledge base. The doc documents in the knowledge base are input into the LLM model to generate the first QA pair including the question and answer. After confirming the QA pair, it is recorded as a qa document type and saved to the knowledge base. The real question is rewritten using the RAG pattern to generate the real answer to the real question. After confirming the rewritten question and the real answer, it is set as the second QA pair and saved to the knowledge base. The document type of the second QA pair is recorded as qa type.

[0036] In an optional embodiment, the first strategy module includes:

[0037] The desensitization module is used to set matching keywords for sensitive words in user questions to determine whether to filter them, and to return a safe answer for user questions that need to be filtered; for identity recognition and casual conversation type recognition in user questions, a knowledge base is used to calculate the distance of the semantic information of the user questions in the vector space, and a recognition threshold is set to determine whether to filter them, and to return a safe answer for user questions that need to be filtered.

[0038] In an optional embodiment, the first strategy module includes:

[0039] The first strategy submodule is used to obtain a recall fragment of the user question from the QA class documents in the knowledge base based on the user question using a recall algorithm. The recall fragment includes the question, the answer, and the question and the answer.

[0040] The KMS threshold configuration is fine-tuned based on the recall accuracy and recall rate to become the first threshold under the first recall strategy, and the recalled fragments exceeding the first threshold are selected as the first target answer of the first recall strategy.

[0041] In an optional embodiment, the second strategy module includes:

[0042] The second strategy submodule is used to obtain the top k recall fragments of the first target answer and set them as the first context. The first context is concatenated with the user question and then input into the LLM model to generate the second target answer. The F1 score and prompt words are set as the second threshold for the second target answer. The second threshold is used to perform validity detection and illusion detection on the second target answer.

[0043] In an optional embodiment, the third strategy module includes:

[0044] The third strategy submodule is used to rewrite the user question into k new query questions based on the step-back strategy, hyde strategy, and description transformation strategy. It then uses QA and DOC documents from the knowledge base to recall the query questions. The recalled content is precisely ranked using a ranking fusion method to obtain the top k recalled content. This precisely ranked top k recalled content is then concatenated with the user question as context and input into the LLM model to generate the third target answer. Finally, an F1 score and prompt words are set as a third threshold for the third target answer, and the validity and illusion detection are performed on the third target answer using this third threshold.

[0045] In an optional embodiment, the third strategy module includes:

[0046] The fourth strategy submodule is used to adjust the LLM model's adherence to the context of the top k recalled content if the third target answer does not meet the third threshold, and to re-enter the user question into the LLM model to generate the fourth target answer.

[0047] A third aspect of this invention discloses a multi-strategy intelligent question-answering device, comprising:

[0048] At least one processor, and,

[0049] A memory communicatively connected to the at least one processor; wherein,

[0050] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform a multi-strategy intelligent question-answering method as disclosed in any of the first aspects of the present invention.

[0051] A fourth aspect of the present invention discloses a computer-readable storage medium storing computer-executable instructions for causing a computer to perform a multi-strategy intelligent question-answering method as disclosed in any of the first aspects of the present invention.

[0052] Compared with the prior art, the present invention has the following advantages:

[0053] This invention proposes a multi-strategy intelligent question-answering method, system, device, and medium. The method includes: acquiring knowledge documents and processing real questions into different document types to construct a knowledge base; filtering user questions based on de-identification detection; returning a first target answer that meets a first threshold based on a first recall strategy; if the first target answer does not meet the first threshold, updating the user question based on the first target answer and executing a second recall strategy to return a second target answer that meets the second threshold; if the second target answer does not meet the second threshold, rewriting the user question based on the knowledge base and executing a third recall strategy to return a third target answer to the user question. This method combines the advantages of FAQ and RAG models, constructs a high-quality knowledge base by generating documents, and dynamically uses different recall strategies for questions of varying difficulty. Especially when the knowledge base cannot answer a question, it can proactively allow the model to use other knowledge to answer. This significantly improves the recall effect while ensuring the performance of the question-answering system, while reducing the overall system response time and improving the overall response efficiency. It has high application value in the field of intelligent question answering. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a flowchart of the multi-strategy intelligent question answering method of the present invention;

[0056] Figure 2 This is a schematic diagram of the multi-strategy intelligent question-answering system of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0058] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0059] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0060] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0061] Example 1

[0062] See Figure 1 This invention discloses a multi-strategy intelligent question-answering method, comprising:

[0063] 101. Acquire knowledge documents and real-world problem-solving materials, categorize them into different document types, and construct a knowledge base;

[0064] It should be noted that knowledge documents typically include, but are not limited to, historical customer consultation records, product manuals, and official documents—files containing knowledge attributes. These are primarily used to build a knowledge base that references the knowledge needed to answer questions. Real-world questions often include questions related to the same professional knowledge to expand the knowledge base's reserves. To further improve the upper limit of the standard answer and the robustness of the knowledge base, knowledge is stored using different document types, facilitating the subsequent return of the standard answer corresponding to the input question by calculating similarity. The intelligent question-answering system in this application primarily addresses professional knowledge questions, such as those in the CRM field. Therefore, the collected knowledge documents and real-world questions will be processed in a CRM-related manner.

[0065] In an optional embodiment, the acquisition of knowledge documents and processing of real-world problems into different document types to construct a knowledge base includes:

[0066] After parsing the acquired knowledge documents, they are divided into blocks. The semantic information of each knowledge block is encoded to generate a knowledge vector. The knowledge blocks and knowledge vectors are combined and recorded as doc documents and saved to the knowledge base.

[0067] Input the doc documents in the knowledge base into the LLM model to generate the first QA pair including questions and answers. After confirming the QA pair, record it as a qa document type and save it to the knowledge base.

[0068] The real question is rewritten using the RAG model, and the real answer to the real question is generated. After confirming the rewritten question and the real answer, it is set as the second QA pair and saved to the knowledge base. The document type of the second QA pair is recorded as qa type.

[0069] Specifically, each knowledge block can be encoded using an embedding model. This involves representing each data point in the knowledge block as a vector, encoding the speech information of the data into a vector space, and capturing the correlation and similarity between data through vectors. This makes similar data closer together in the vector space. The knowledge documents processed by embedding are saved as doc documents to distinguish them from other data in the knowledge base, making it easier to select suitable data when recalling user questions later.

[0070] Furthermore, in addition to using the doc documents generated from the knowledge documents, the LLM (Large Language Model) is also input into the doc documents as training data. The LLM model generates more related questions and answers that can be used as knowledge references. The generalization ability of the knowledge base is improved through the LLM model, and the robustness of intelligent question answering is enhanced. The corresponding questions and answers are saved as QA type documents in the form of QA, which are different from doc documents and facilitate the subsequent retrieval process of calling data in the knowledge base.

[0071] Furthermore, the real questions are those frequently asked in the professional field, obtained from historical data, reports, or surveys, and are not generated by the LLM model. The processing of real questions follows the approach of intelligent question-answering systems in the RAG model: the real question is rewritten into several basic or simple questions. For each simple and basic question, the top k fragments are retrieved from the knowledge base as the context of the real question. This context is then concatenated with the real question and input into the LLM model to obtain the real answer. Since the real answer is derived from the real question, the rewritten question and the real answer are added to the knowledge base as a QA pair and recorded as a QA type document to distinguish it from other documents. This expansion of multiple knowledge bases minimizes the overhead of building and maintaining the knowledge base, increases the range of answers covered by the knowledge base, and enhances the robustness of the intelligent question-answering system. For simple questions in professional knowledge, compared to consuming significant computational resources through the RAG model, some basic questions can be answered first using the knowledge base.

[0072] Furthermore, the various document types mentioned above can be categorized and managed using the Elasticsearch component. Specifically, Elasticsearch can preprocess questions, primarily through tokenization and part-of-speech tagging. It also provides a rich Query DSL (Domain-Specific Language), allowing for the construction of complex queries to achieve precise keyword matching and deep semantic understanding within the question. This helps the question-answering system accurately understand the user's intent and needs, thus providing more precise answers. For different document types, such as doc and qa documents, Elasticsearch adds tags to questions based on their content and business context, facilitating the categorization and management of question-answer pairs. Elasticsearch supports custom fields and index types, easily enabling tag categorization and management. By adding tags to questions, similar questions can be matched within the tagged categories, improving machine understanding and matching accuracy. This allows the question-answering system to handle large-scale data and respond quickly to user needs.

[0073] 102. Filter user questions based on desensitization detection, and return the first target answer that meets the first threshold based on the first recall strategy;

[0074] It should be noted that the intelligent question-answering system primarily addresses professional knowledge questions, such as those related to CRM. Therefore, the collected knowledge documents and real-world questions will be processed around the CRM domain. To avoid unnecessary resource consumption, when receiving user input, anonymization checks are performed first. Specific checks include, but are not limited to, sensitive word detection, identity verification, and casual conversation detection. This is mainly used to distinguish whether the user's purpose is to query answers to professional knowledge questions. For filtered user questions, a first recall strategy is used to filter simpler questions, which can return partial answers and further determine whether a different strategy is needed to differentiate the complexity of user questions, thereby reducing computational overhead and improving response efficiency.

[0075] In an optional embodiment, filtering user questions based on desensitization detection includes:

[0076] Set matching keywords for sensitive words in user questions to determine whether to filter them, and return a safe answer for user questions that need to be filtered;

[0077] For user questions, the knowledge base is used to calculate the distance between the semantic information of the user questions in the vector space for identity recognition and casual conversation type recognition. A recognition threshold is set to determine whether to filter. For user questions that need to be filtered, a safe answer is returned.

[0078] Specifically, for sensitive words, keywords containing sensitive words can be filtered. For identity verification and casual conversation questions, the purpose of the user's question can be determined through QA in the knowledge base. If it is for identity verification (e.g., asking: Is the QA system the OPENAI system?), then to protect privacy, this type of question will not be answered. If it is a casual conversation question, to reduce the consumption of computing resources, only questions related to professional knowledge will be answered. In both cases, a safe answer will be returned, for example (I am a robot with CRM professional knowledge, and I cannot answer your question. Please ask other CRM-related questions).

[0079] In an optional embodiment, returning the first target answer that satisfies the first threshold based on the first recall strategy includes:

[0080] Based on the user's question, a recall algorithm is used to obtain a recall fragment that matches the user's question in the QA category documents in the knowledge base. The recall fragment includes the question, the answer, and the question and the answer.

[0081] The KMS threshold configuration is fine-tuned based on the recall accuracy and recall rate to become the first threshold under the first recall strategy, and the recalled fragments exceeding the first threshold are selected as the first target answer of the first recall strategy.

[0082] Specifically, for the first received user question, a preliminary answer is provided by recalling the corresponding answer from the knowledge base, which can be understood as a simple answer. The first threshold can be set not only through KMS thresholding but also through rules to filter the recalled content, selecting the standard answer that matches the user question. Since the knowledge base distinguishes different document types during setup, QA type documents can establish a connection with the user question more quickly compared to other knowledge document types, thus rapidly responding and returning the standard answer. This eliminates the need to judge whether the strategy meets the user question's requirements; instead, it prioritizes generating the corresponding answer and then examines its reasonableness, reducing unnecessary computational overhead and improving the performance of intelligent question answering. If the returned answer fails to meet the user question, it indicates that the user question is not simple, and other in-depth strategies are needed to return answers to more complex questions.

[0083] 103. If the first target answer does not meet the first threshold, then update the user question according to the first target answer, execute the second recall strategy, and return the second target answer that meets the second threshold;

[0084] In an optional embodiment, the step of updating the user question based on the first target answer and executing a second recall strategy to return a second target answer that meets the second threshold if the first target answer does not meet the first threshold includes:

[0085] The top k recalled fragments of the first target answer are set as the first context. The first context is concatenated with the user question and then input into the LLM model to generate the second target answer.

[0086] The F1 score and the cue word are set as the second threshold for the second target answer. The validity and illusion detection of the second target answer are performed by using the second threshold.

[0087] Specifically, if the first strategy fails to generate an answer to a user's question, it's possible that the user's question is not a simple question but rather consists of, but is not limited to, multiple simple questions, or lacks context. Therefore, when the first target answer is not met, the top k recall fragments returned by the first strategy can serve as supplementary knowledge bases for the user's question context or multiple simple questions. These top k recall fragments are then inserted into the user's question as context. Context is a crucial concept in question-answering systems, helping the system to more accurately understand user intent during queries and thus provide more accurate answers. In this embodiment, the context further enriches the description of the user's question, facilitating the LLM model's understanding of user intent. The LLM model itself can learn from the training corpus and external knowledge bases to obtain the most relevant text content for the user's question, and finally combines the relevant content with the question to generate the final answer. This solves the difficult questions that the first recall strategy cannot answer. The second recall strategy uses the recall fragments generated by the first strategy, making the question closer to the user's true intent. This process also reduces the computational resource consumption of the question-answering system and improves system performance.

[0088] Furthermore, the second threshold is primarily used to detect the feasibility of the second target answer and whether the answer generated by the LLM model exhibits model illusion. The F1 score mainly includes the correctness (evaluating the true accuracy of the model's generated response), completeness (measuring the completeness of the types of output information the model can provide based on user input), relevance (characterizing the correlation between the output information of the model's response based on a series of related user inputs), and effectiveness (evaluating the usefulness of the model's generated response to user input). Specific scoring methods, such as the correctness score, are implemented by comparing it with known facts or standard answers. Metrics are typically categorized into multiple levels, such as a 6-level rating system with a maximum score of 5, or the harmonic mean of precision and recall to estimate the model's accuracy in tasks like classification, extraction, and error correction. For the illusion problem in LLM models, prompts can be used, such as asking whether the large language model generates answers from incomplete or ambiguous contexts, or static prompt testing can be employed. This involves creating a set of prompts relevant to the application scenario and benchmarking the model's response. By manually evaluating the correctness and coherence of the output, the presence of illusion problems in the model can be identified. Combining these detection methods can improve the accuracy and reliability of the model's output answers and reduce the impact of model illusions on the generated answers.

[0089] 104. If the second target answer does not meet the second threshold, then rewrite the user question according to the knowledge base and execute the third recall strategy to return the third target answer of the user question.

[0090] It should be noted that if the second target answer fails to meet the second threshold, there may be some problems. For example, in the second recall strategy, the large language model may focus too much on a basic or simple question in the user's question after splicing the context, which may result in an unsatisfactory second target answer. Therefore, it is necessary to continue to adjust the recall strategy.

[0091] In an optional embodiment, the step of rewriting the user question according to the knowledge base and executing a third recall strategy to return the third target answer to the user question if the second target answer does not meet the second threshold includes:

[0092] The user question is rewritten into k new query questions based on the step-back strategy, hyde strategy, and description transformation strategy. The query questions are recalled using QA and DOC documents in the knowledge base. The recalled content is ranked and fused to obtain the top k recalled content. The top k recalled content is concatenated with the user question as context and input into the LLM model to generate the third target answer.

[0093] The F1 score and the prompt word are set as the third threshold for the third target answer. The validity and illusion detection of the third target answer are performed by using the third threshold.

[0094] Specifically, to enable the LLM (Large Language Model) to better understand user questions and reduce its focus on a single question, a step-back approach is used. This approach reduces the user question to a simplified version that is easier for the LLM to understand, thus improving the solution. For example, before posing the user question, a more basic or intuitive question is presented to help the LLM view and understand the original question from different perspectives. Therefore, the initial user input is rewritten into k query questions after step-back processing. These queries are then used as input to retrieve information from various document types in the knowledge base (such as QA and DOC documents) to obtain as much content as possible to support the solutions to the query questions after step-back. Hypothetical document embeddings (HDE) are used to treat hypothetical documents as "hypothetical answers" to the queries, generating richer context with hypothetical attributes, thereby expanding the quantity and quality of the query questions. A description transformation strategy is used to transform the wording of the generated query questions, generating different descriptions for query questions with the same meaning to expand the query question data and facilitate model understanding. Further, Reciprocal Radiometry (RRF) is employed. Rank fusion combines and sorts the recall results obtained from queries of different document types. Since the RRF ranking fusion method can take into account the position of the recalled content in the original ranking (i.e., the ranking of the top k recalled content returned in different document types), ranking fusion will give higher-ranked recalled content in different result lists greater importance, thereby effectively combining the results of multiple retrieval machines to improve the overall performance and accuracy of the question answering system. Since the second recall strategy cannot obtain a satisfactory answer, the computational cost of the system can be appropriately sacrificed in order to improve the effectiveness and accuracy of the output answer. However, in this method, user questions are processed hierarchically using multiple strategies. Therefore, unless a third recall strategy is necessary, other recall strategies can solve most user questions, which reduces the computational cost for the question answering system as a whole. In practice, the output of the previous strategy also provides reference knowledge for the input of the next strategy, avoiding the resource consumption caused by repeated calculations. Finally, the top k recalled content after fusion and ranking is concatenated with the user question and input into the LLM large language model to generate the third target answer (i.e. the answer to the difficult question).

[0095] Furthermore, in the above steps, it is also necessary to perform validity testing and illusion testing on the large language model and its output results. It is understandable that, apart from further adjusting the weight of the evaluation score and the rules of the prompt words, the detection process of the third recall strategy is roughly the same as that of the second recall strategy.

[0096] In an optional embodiment, the step of rewriting the user question according to the knowledge base and executing a third recall strategy to return the third target answer to the user question if the second target answer does not meet the second threshold further includes:

[0097] If the third target answer does not meet the third threshold, the LLM model's adherence to the context of the top k recalled content is adjusted, and the user question is re-entered into the LLM model to generate the fourth target answer.

[0098] Specifically, in order to limit the output of large language models from being detached from the user's question, some rules that need to be followed are usually pre-set before the user's question is input. If the third recall strategy still cannot generate a satisfactory answer to the user's question, the rules for following the recall content can be relaxed. This can be understood as increasing the range of optional recall content, thereby providing the user with a satisfactory answer from more options. It can be understood that in the above steps, apart from adjusting the model's adherence to the restrictions, the other steps are consistent with the third recall strategy. However, because the adherence restrictions are relaxed, the actual generated answers will be richer, thereby outputting answers that are as satisfactory to the user's question as possible.

[0099] This invention proposes a multi-strategy intelligent question-answering method, system, device, and medium. The method includes: acquiring knowledge documents and processing real questions into different document types to construct a knowledge base; filtering user questions based on de-identification detection; returning a first target answer that meets a first threshold based on a first recall strategy; if the first target answer does not meet the first threshold, updating the user question based on the first target answer and executing a second recall strategy to return a second target answer that meets the second threshold; if the second target answer does not meet the second threshold, rewriting the user question based on the knowledge base and executing a third recall strategy to return a third target answer to the user question. This method combines the advantages of FAQ and RAG models, constructs a high-quality knowledge base by generating documents, and dynamically uses different recall strategies for questions of varying difficulty. Especially when the knowledge base cannot answer a question, it can proactively allow the model to use other knowledge to answer. This significantly improves the recall effect while ensuring the performance of the question-answering system, while reducing the overall system response time and improving the overall response efficiency. It has high application value in the field of intelligent question answering.

[0100] like Figure 2 As shown, a second aspect of the present invention discloses a multi-strategy intelligent question-answering system, the system comprising:

[0101] The knowledge base module is used to acquire knowledge documents and real-world problem solutions, categorized into different document types to build the knowledge base.

[0102] The first strategy module is used to filter user questions based on de-identification detection and return the first target answer that meets the first threshold based on the first recall strategy.

[0103] The second strategy module is used to update the user question based on the first target answer and execute the second recall strategy if the first target answer does not meet the first threshold, and return the second target answer that meets the second threshold.

[0104] The third strategy module is used to rewrite the user question based on the knowledge base and execute the third recall strategy if the second target answer does not meet the second threshold, returning the third target answer to the user question.

[0105] In an optional embodiment, the knowledge base module includes:

[0106] The knowledge base submodule is used to parse and segment the acquired knowledge documents, encode the semantic information of each knowledge block to generate a knowledge vector, and combine the knowledge block and knowledge vector into a doc document type and save it to the knowledge base. The doc documents in the knowledge base are input into the LLM model to generate the first QA pair including the question and answer. After confirming the QA pair, it is recorded as a qa document type and saved to the knowledge base. The real question is rewritten using the RAG pattern to generate the real answer to the real question. After confirming the rewritten question and the real answer, it is set as the second QA pair and saved to the knowledge base. The document type of the second QA pair is recorded as qa type.

[0107] In an optional embodiment, the first strategy module includes:

[0108] The desensitization module is used to set matching keywords for sensitive words in user questions to determine whether to filter them, and to return a safe answer for user questions that need to be filtered; for identity recognition and casual conversation type recognition in user questions, a knowledge base is used to calculate the distance of the semantic information of the user questions in the vector space, and a recognition threshold is set to determine whether to filter them, and to return a safe answer for user questions that need to be filtered.

[0109] In an optional embodiment, the first strategy module includes:

[0110] The first strategy submodule is used to obtain a recall fragment of the user question from the QA class documents in the knowledge base based on the user question using a recall algorithm. The recall fragment includes the question, the answer, and the question and the answer.

[0111] The KMS threshold configuration is fine-tuned based on the recall accuracy and recall rate to become the first threshold under the first recall strategy, and the recalled fragments exceeding the first threshold are selected as the first target answer of the first recall strategy.

[0112] In an optional embodiment, the second strategy module includes:

[0113] The second strategy submodule is used to obtain the top k recall fragments of the first target answer and set them as the first context. The first context is concatenated with the user question and then input into the LLM model to generate the second target answer. The F1 score and prompt words are set as the second threshold for the second target answer. The second threshold is used to perform validity detection and illusion detection on the second target answer.

[0114] In an optional embodiment, the third strategy module includes:

[0115] The third strategy submodule is used to rewrite the user question into k new query questions based on the step-back strategy, hyde strategy, and description transformation strategy. It then uses QA and DOC documents from the knowledge base to recall the query questions. The recalled content is precisely ranked using a ranking fusion method to obtain the top k recalled content. This precisely ranked top k recalled content is then concatenated with the user question as context and input into the LLM model to generate the third target answer. Finally, an F1 score and prompt words are set as a third threshold for the third target answer, and the validity and illusion detection are performed on the third target answer using this third threshold.

[0116] In an optional embodiment, the third strategy module includes:

[0117] The fourth strategy submodule is used to adjust the LLM model's adherence to the context of the top k recalled content if the third target answer does not meet the third threshold, and to re-enter the user question into the LLM model to generate the fourth target answer.

[0118] A third aspect of this invention discloses a multi-strategy intelligent question-answering device, comprising:

[0119] At least one processor, and,

[0120] A memory communicatively connected to the at least one processor; wherein,

[0121] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform a multi-strategy intelligent question-answering method as disclosed in any of the first aspects of the present invention.

[0122] The computer device can be a terminal, and includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a multi-strategy intelligent question-answering method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0123] A fourth aspect of the present invention discloses a computer-readable storage medium storing computer-executable instructions for causing a computer to perform a multi-strategy intelligent question-answering method as disclosed in any of the first aspects of the present invention.

[0124] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the multi-strategy intelligent question-answering method described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0125] Alternatively, if the above-mentioned modules of the present invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present invention, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, terminal, or network device, etc.) to execute all or part of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, RAM, ROM, magnetic disks, or optical disks.

[0126] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-strategy intelligent question-answering method, characterized in that, The method comprises: acquiring knowledge documents and real questions and processing them into different document types to build a knowledge base; filtering the user question according to desensitization detection, returning a first target answer meeting a first threshold based on a first recall strategy; if the first target answer does not meet the first threshold, performing a second recall strategy on the user question according to the first target answer, and returning a second target answer meeting a second threshold; if the second target answer does not meet the second threshold, performing a third recall strategy on the user question according to the knowledge base, and returning a third target answer of the user question; if the first target answer does not meet the first threshold, performing a second recall strategy on the user question according to the first target answer, and returning a second target answer meeting a second threshold, comprising: acquiring topk recall segments of the first target answer and setting them as a first context, splicing the first context with the user question to input an LLM model to generate a second target answer; setting F1 score and prompt words as the second threshold to detect the effectiveness and illusion of the second target answer through the second threshold; if the second target answer does not meet the second threshold, performing a third recall strategy on the user question according to the knowledge base, and returning a third target answer of the user question, comprising: rewriting the user question into k new query questions according to a step-back strategy, a hyde strategy and a description conversion strategy, recalling the query questions using qa documents and doc documents in the knowledge base, accurately sorting the recall contents by a ranking fusion method to obtain topk recall contents, and splicing the topk accurately sorted recall contents as a context to the user question to input an LLM model to generate a third target answer; setting F1 score and prompt words as the third threshold to detect the effectiveness and illusion of the third target answer through the third threshold.

2. The multi-strategy intelligent question answering method of claim 1, wherein, The method comprises: analyzing the acquired knowledge documents and processing them in blocks, encoding semantic information of each knowledge block to generate a knowledge vector, combining the knowledge block and the knowledge vector to record as a doc document type and save to the knowledge base; inputting the doc documents in the knowledge base into an LLM model to generate a first QA pair comprising a question and an answer, confirming the QA pair to record as a qa document type and save to the knowledge base; rewriting the question content of the real question through a RAG mode and generating a real answer of the real question, confirming the rewritten question and the real answer to set as a second QA pair and save to the knowledge base, and recording the document type of the second QA pair as qa type.

3. The multi-strategy intelligent question answering method of claim 1, wherein, The method comprises: setting a matching keyword in the sensitive word in the user question to determine whether to filter, and returning a safe answer for the user question that needs to be filtered; calculating the semantic information of the user question in the vector space using the knowledge base to set a recognition threshold to determine whether to filter, and returning a safe answer for the user question that needs to be filtered.

4. The multi-strategy intelligent question-answering method of claim 1, wherein, The first target answer meeting the first threshold value is returned based on the first recall strategy, and the first target answer meeting the second threshold value is returned based on the second recall strategy. According to the user question, the recall algorithm is used to obtain the recall segment matching the user question in the QA type document of the knowledge base, and the recall segment includes the question, the answer, and the question and the answer; The KMS threshold value is fine-tuned as the first threshold value under the first recall strategy according to the recall accuracy and the recall rate, and the recall segment exceeding the first threshold value is selected as the first target answer of the first recall strategy.

5. The multi-strategy intelligent question-answering method of claim 1, wherein, If the second target answer does not meet the second threshold value, a third recall strategy is executed according to the rewritten user question in the knowledge base, and a third target answer of the user question is returned. If the third target answer does not meet the third threshold value, the following restriction of the LLM model on the topk recall content contexts is adjusted, and the fourth target answer is generated by re-inputting the user question into the LLM model.

6. A multi-strategy intelligent question answering system, characterized in that, The system comprises: The knowledge base module is used to acquire knowledge documents and real question processing to construct a knowledge base in different document types; The first strategy module is used to filter the user question according to the desensitization detection, and return the first target answer meeting the first threshold value based on the first recall strategy; The second strategy module is used to execute the second recall strategy according to the updated user question of the first target answer if the first target answer does not meet the first threshold value, and return the second target answer meeting the second threshold value; The third strategy module is used to execute the third recall strategy according to the rewritten user question in the knowledge base if the second target answer does not meet the second threshold value, and return the third target answer of the user question. If the first target answer does not meet the first threshold value, the second recall strategy is executed according to the updated user question of the first target answer, and the second target answer meeting the second threshold value is returned, comprising: The topk recall segments of the first target answer are set as the first context, and the second target answer is generated by inputting the first context and the user question into the LLM model; The F1 score and the prompt word are set as the second threshold value for the second target answer, and the effectiveness detection and the illusion detection are performed on the second target answer through the second threshold value; If the second target answer does not meet the second threshold value, the third recall strategy is executed according to the rewritten user question in the knowledge base, and the third target answer of the user question is returned, comprising: The user question is rewritten into k new query questions according to the step-back strategy, the hyde strategy and the description conversion strategy, the query question is recalled by using the QA document and the doc document in the knowledge base, the topk recall contents are obtained by using the accurate sorting method for the recall content, and the third target answer is generated by inputting the topk recall contents after the accurate sorting into the LLM model. The F1 score and the prompt word are set as the third threshold value for the third target answer, and the effectiveness detection and the illusion detection are performed on the third target answer through the third threshold value.

7. A multi-strategy intelligent question answering device, characterized in that, It comprises: At least one processor, and The memory is connected in communication with the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the multi-policy intelligent question-answering method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions for causing a computer to perform the multi-policy intelligent question-answering method according to any one of claims 1 to 5.

Images