Retrieval enhanced generated knowledge base optimization method and system

Abstract

The application provides a retrieval enhanced generated knowledge base optimization method and system, relates to the technical field of artificial intelligence and natural language processing, calculates the comprehensive feedback score of a user on a question and answer pair by using various user feedback data, determines the feedback category of the question and answer pair according to the comprehensive feedback score and the user feedback data, executes the knowledge base operation strategy corresponding to the current feedback category, realizes the low-cost and efficient optimization of the RAG system, and due to the setting of the corresponding knowledge base operation strategy for each feedback category, the RAG system is accurately optimized, and the reliability of the query result is improved. Secondly, by adding the instances of the positive feedback category and the instances of the negative feedback category into the positive knowledge set and the negative knowledge set respectively, the negative knowledge set is used to actively avoid repeating the same mistake when responding to a new user query, and the reliability of the query result is further improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing technology, specifically to a method and system for optimizing a knowledge base generated through retrieval enhancement. Background Technology

[0002] Currently, LLM (Large Language Model) has become a key driving force for the development of natural language processing applications. However, because the internal knowledge of an LLM is entirely derived from the dataset used during its training, once training is complete, its internal knowledge will not change, resulting in a time lag in the internal knowledge of an LLM, which cannot reflect the latest information changes. Furthermore, without external information source constraints, an LLM may "fabricate" an answer that best fits the context based on statistical patterns in its training data, which may lead to the output of non-factual content. Moreover, the knowledge system of an LLM is usually general-domain, lacking in-depth coverage of specific industries or private domains.

[0003] To overcome the limitations of LLM (Low-Level Machine Learning), Retrieval-Augmented Generation (RAG) was proposed. The core mechanism of RAG lies in that, when responding to user requests, it doesn't directly answer using the LLM. Instead, it first retrieves authoritative information highly relevant to the user's request from an external knowledge base, and then injects this information as real-time context into the LLM to guide the generation of accurate and evidence-based responses. The introduction of the RAG framework significantly improves the factual consistency and timeliness of content generated by the LLM. However, after deploying a RAG system, the content of the RAG knowledge base is typically static. This leads to systematic and repetitive question-and-answer failures when errors or gaps exist in the knowledge base, and the timeliness and completeness of the knowledge content continuously decrease.

[0004] The main optimization approach in existing technologies is to fine-tune the LLM model itself. However, this approach has extremely high requirements for computing resources and high-quality labeled data, resulting in high costs. Furthermore, this approach has long training and deployment cycles, making it unable to quickly respond to knowledge gaps discovered online. Moreover, the effectiveness of fine-tuning is uncertain, making it difficult to accurately and controllably correct specific errors in the knowledge base, leading to low reliability of user query results. Summary of the Invention

[0005] In view of this, a method and system for optimizing the knowledge base generated by retrieval enhancement are provided to solve the problems of high cost, low efficiency and low reliability of user query results in the optimization of RAG systems in the existing technology.

[0006] The present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for optimizing a knowledge base generated with enhanced retrieval capabilities, comprising:

[0008] Acquire user feedback data associated with question-answer pairs generated by the search enhancement generation (RAG) system, and use the user feedback data to calculate a comprehensive feedback score for the user on the question-answer pairs; the user feedback data includes at least the quantitative score, natural language comments, and implicit feedback behavior data; implicit feedback behavior is non-user-initiated feedback behavior that can reflect whether the user is satisfied with the current query results.

[0009] Based on the comprehensive feedback score and the user feedback data, the feedback category of the question-and-answer pair is determined; the feedback category is divided into positive feedback category and negative feedback category;

[0010] Based on the feedback category, a target knowledge base operation strategy is selected from a set of preset knowledge base operation strategies, and the target knowledge base operation strategy is executed. Specifically, when the feedback category is the positive feedback category, the target data structure is added to the positive knowledge set; when the feedback category is the negative feedback category, the target data structure is added to the negative knowledge set. The target data structure is the question-answer pair or its enhanced version. The negative knowledge set is used, in response to a new user query, to penalize the original retrieval score of the query result based on the semantic similarity between the new user query and each original user question in the negative knowledge set, and then filter the query results based on the penalized retrieval score.

[0011] Optionally, using the user feedback data, a comprehensive feedback score for the user's response to the question-and-answer pair is calculated, specifically including:

[0012] When the user feedback data includes the natural language comments, sentiment analysis is performed on the natural language comments to quantify them into a sentiment score; when the user feedback data does not include the natural language comments, the sentiment score is determined to be 0.

[0013] The implicit feedback behavior represented by the implicit feedback behavior data is analyzed to determine whether the implicit feedback behavior is an implicit negative feedback behavior, and a behavior signal is generated based on the determination result; wherein, when the implicit feedback behavior is an implicit negative feedback behavior, the behavior signal is 1, otherwise the behavior signal is 0.

[0014] The quantitative score, the emotional score, and the behavioral signal are normalized.

[0015] The normalized quantitative score, the emotional score, and the behavioral signal are weighted and calculated to obtain the comprehensive feedback score.

[0016] Optionally, the positive feedback category is specifically divided into a positive feedback category without natural language comments and a positive feedback category with natural language comments;

[0017] The negative feedback categories are specifically divided into two categories: negative feedback without natural language comments and negative feedback with natural language comments.

[0018] Optionally, the feedback category of the question-and-answer pair is determined based on the comprehensive feedback score and the user feedback data, specifically including:

[0019] When the overall feedback score is greater than or equal to a preset score threshold, and the user feedback data does not contain the natural language comments, the feedback category is determined to be the positive feedback and no natural language comments category.

[0020] When the overall feedback score is greater than or equal to a preset score threshold, and the user feedback data contains the natural language comments, the feedback category is determined to be the positive feedback category with natural language comments.

[0021] When the overall feedback score is less than a preset score threshold and the user feedback data contains the natural language comments, the feedback category is determined to be the negative feedback category with natural language comments.

[0022] When the overall feedback score is less than a preset score threshold and the user feedback data does not contain the natural language comments, the feedback category is determined to be the negative feedback and no natural language comments category.

[0023] Optionally, when the feedback category is positive feedback and there is a natural language comment category, the target knowledge base operation strategy includes:

[0024] Using a preset language model, the answers in the question-answer pair are augmented based on the natural language comments to generate an enhanced version of the question-answer pair;

[0025] The enhanced version of the question-and-answer pair is added to the positive knowledge set.

[0026] Optionally, when the feedback category is negative feedback and there is a natural language comment category, the target knowledge base operation strategy includes:

[0027] The confidence weight of the question-answer pair is set to the first preset weight;

[0028] Using a pre-defined language model, the natural language comments and the question-answer pairs are analyzed to generate an error analysis report; the enhanced version of the question-answer pairs includes the question-answer pairs, their error analysis reports, and confidence weights.

[0029] The enhanced version of the question-answer pair is added to the negative knowledge set.

[0030] Optionally, when the feedback category is negative feedback and there is no natural language comment category, the target knowledge base operation strategy includes:

[0031] The confidence weight of the question-answer pair is set to the second preset weight;

[0032] The question-answer pairs assigned confidence weights are added to the negative knowledge set.

[0033] Optionally, when the feedback category is positive feedback and there is no natural language comment category, the target knowledge base operation strategy includes:

[0034] Add the question-and-answer pairs to the positive knowledge set.

[0035] Optionally, the positive knowledge set is a positive knowledge base;

[0036] The set of negative knowledge forms an anti-knowledge base;

[0037] The positive knowledge base and the negative knowledge base are logically or physically separated from each other.

[0038] Secondly, the present invention also provides a knowledge base optimization system for retrieval enhancement generation, comprising:

[0039] The feedback receiving module is used to acquire user feedback data associated with question-answer pairs generated by the retrieval enhancement generation (RAG) system, and to calculate a comprehensive feedback score for the user on the question-answer pairs using the user feedback data. The user feedback data includes at least the quantitative score and the implicit feedback behavior data from quantitative scores, natural language comments, and implicit feedback behavior data. Implicit feedback behavior refers to non-user-initiated feedback behavior that reflects whether the user is satisfied with the current query results.

[0040] The feedback classification module is used to determine the feedback category of the question-and-answer pair based on the comprehensive feedback score and the user feedback data; the feedback category is divided into positive feedback category and negative feedback category;

[0041] The knowledge base operation module is used to select a target knowledge base operation strategy from a variety of preset knowledge base operation strategies according to the feedback category, and execute the target knowledge base operation strategy; wherein, when the feedback category is the positive feedback category, the target data structure is added to the positive knowledge set; when the feedback category is the negative feedback category, the target data structure is added to the negative knowledge set; the target data structure is the question-answer pair or its enhanced version; wherein, when responding to a new user query, the negative knowledge set is used to penalize the original retrieval score of the query result of the new user query based on the semantic similarity between the new user query and each original user question in the negative knowledge set, and then filter the query results according to the penalized retrieval score.

[0042] This invention employs the above technical solution, utilizing various user feedback data (quantitative ratings, natural language comments, and implicit feedback behavior data) to calculate a comprehensive feedback score for user responses to question-and-answer pairs. Based on the comprehensive feedback score and user feedback data, the feedback category of the question-and-answer pair is determined, and a knowledge base operation strategy corresponding to the current feedback category is executed. This direct and automated editing of the knowledge base text content achieves low-cost and high-efficiency optimization of the RAG system. Furthermore, by fully utilizing various user feedback data and further subdividing feedback categories based on the comprehensive feedback score and user feedback data, corresponding knowledge base operations are set for each feedback category. The strategy achieves precise optimization of the RAG system, improving the reliability of query results. Secondly, by adding instances of positive feedback categories and instances of negative feedback categories to the positive knowledge set and negative knowledge set respectively, the negative knowledge set is used to penalize the original retrieval score of the query results of the new user query based on the semantic similarity between the new user query and each original user question in the negative knowledge set when responding to a new user query. Then, the query results are filtered according to the penalized retrieval score, enabling this application to proactively avoid known errors and further improve the reliability of query results. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating a method for optimizing a knowledge base generated with enhanced retrieval capabilities, as provided in an embodiment of the present invention.

[0045] Figure 2 This is a schematic diagram of the structure of a knowledge base optimization system for enhanced retrieval provided in an embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0047] Figure 1 This is a flowchart illustrating a method for optimizing a knowledge base through enhanced retrieval provided in an embodiment of the present invention. For example... Figure 1 As shown, this process includes:

[0048] Step 101: Obtain user feedback data associated with the question-answer pairs generated by the search enhancement generation RAG system, and use the user feedback data to calculate the user's comprehensive feedback score for the question-answer pairs; the user feedback data includes at least quantitative scores, natural language comments, and implicit feedback behavior data; implicit feedback behavior is non-user-initiated feedback behavior that can reflect whether the user is satisfied with the current query results.

[0049] Specifically, users can provide structured quantitative ratings through the user feedback interface. Quantitative ratings ensure the standardization and comparability of feedback data, providing standardized numerical input for subsequent steps. In a concrete example, the user feedback interface can provide a Likert scale from 1 to 5 stars, which users can directly rate. If a user finds an answer incorrect, they might give a low score of 1 star. The number of stars represents the score; for example, 1 star is 1 point, and 2 stars are 2 points. Furthermore, quantitative ratings can also take other forms, such as binary ratings (e.g., "like" / "dislike"), slider ratings, or preset tags (e.g., "helpful," "incorrect," "incomplete"). In the binary rating format, "like" is worth 5 points, and "dislike" is worth 1 point; slider ratings allow users to select a score within a continuous numerical range (1-5 points) by dragging a slider. In the preset tag format, "helpful" is worth 5 points, "incorrect" is worth 1 point, and "incomplete" is worth 3 points.

[0050] Users can provide unstructured natural language feedback through the user feedback interface. The user feedback interface can provide guided forms, such as "What do you think is incorrect?" or "What information would you like to add?", to guide users in filling out natural language feedback.

[0051] Implicit feedback behavior can be categorized as follows: First, whether a user performs consecutive queries within a short period. If a user performs consecutive queries within a short period, it indicates that the user is dissatisfied with the current query results and has retrieval again; conversely, if the user does not perform consecutive queries, it indicates that the user is satisfied with the current query results. Second, whether a user stays on the results page for more than a preset time. If the user stays on the results page for more than the preset time, it indicates that the user is satisfied with the current query results; otherwise, it indicates dissatisfaction. Third, whether a user copies the query results. If the user copies the query results, it indicates that the user is satisfied with the current query results; otherwise, it indicates dissatisfaction. Fourth, whether a user clicks the system-recommended "next step" link. If the user clicks the system-recommended "next step" link, it indicates that the user is satisfied with the current query results; otherwise, it indicates dissatisfaction. In another example, implicit feedback behavior could also be whether the frequency of a user's clicks in the same area of ​​the results page exceeds a preset frequency threshold. If the user's click frequency in the same area of ​​the results page exceeds the preset frequency threshold (i.e., angrily clicking frequently), it indicates that the user is not satisfied with the query results; otherwise, they are satisfied.

[0052] Step 102: Determine the feedback category of the question-and-answer pair based on the overall feedback score and user feedback data; the feedback categories are divided into positive feedback category and negative feedback category.

[0053] Step 103: Based on the feedback category, select the target knowledge base operation strategy from a variety of preset knowledge base operation strategies and execute the target knowledge base operation strategy; wherein, when the feedback category is positive feedback, add the target data structure to the positive knowledge set; when the feedback category is negative feedback, add the target data structure to the negative knowledge set; the target data structure is a question-answer pair or its enhanced version; wherein, when responding to a new user query, the negative knowledge set is used to penalize the original retrieval score of the query results of the new user query based on the semantic similarity between the new user query and each original user question in the negative knowledge set, and then filter the query results based on the penalized retrieval score.

[0054] In a specific example, when responding to a new user query, firstly, a search is performed in the positive knowledge set, assuming query result 1, query result 2, query result 3, and their original search scores. Then, for the query result with the highest original search score, a penalized adjustment is made to its original search score based on the semantic similarity between the new user query and each original user question in the negative knowledge set, as shown in the following formula:

[0055] ......(1)

[0056] in, This indicates the retrieval score of the current query result (e.g., query result 1) after penalty adjustment; This represents the original search score of the current query result; This indicates a new user query; Indicates the first negative knowledge set The original user problem of the negative example; express and Semantic similarity between them; Indicates the first negative knowledge set Confidence weights for each negative example; This represents the total number of negative examples in the negative knowledge set.

[0057] Finally, The value is compared with the preset score threshold, and in When the value is greater than the preset score threshold, the current query result is output; If the value is not greater than the preset score threshold, a prompt message will be output indicating that the query failed.

[0058] In this way, the probability of new user queries that are similar to those in the negative knowledge set being returned the same incorrect query results can be reduced, thus proactively avoiding repeating the same mistake.

[0059] This invention employs the above technical solution, utilizing various user feedback data (quantitative scores, natural language comments, and implicit feedback behavior data) to calculate a comprehensive feedback score for the question-and-answer pair. Based on the comprehensive feedback score and user feedback data, the feedback category of the question-and-answer pair is determined, and a knowledge base operation strategy corresponding to the current feedback category is executed. This direct and automated editing of the knowledge base text content achieves low-cost and high-efficiency optimization of the RAG system. Furthermore, by fully utilizing various user feedback data and further subdividing feedback categories based on the comprehensive feedback score and user feedback data, a corresponding knowledge base is set up for each feedback category. The operational strategy achieves precise optimization of the RAG system, improving the reliability of query results. Secondly, by adding instances of positive feedback categories and instances of negative feedback categories to the positive knowledge set and negative knowledge set respectively, the negative knowledge set is used to penalize the original retrieval score of the query results of the new user query based on the semantic similarity between the new user query and each original user question in the negative knowledge set when responding to a new user query. Then, the query results are filtered according to the penalized retrieval score, enabling this application to proactively avoid known errors and further improve the reliability of query results.

[0060] In this embodiment of the invention, user feedback data is used to calculate the comprehensive feedback score of the user's response to the question-and-answer pair, which may specifically include:

[0061] (1) When the user feedback data contains natural language comments, perform sentiment analysis on the natural language comments to quantify them into sentiment scores; when the user feedback data does not contain natural language comments, determine the sentiment score as 0.

[0062] Specifically, when user feedback data includes natural language comments, the first step is to perform deep natural language processing on the comments, which may include preprocessing steps such as word segmentation, part-of-speech tagging, and named entity recognition. Then, sentiment analysis algorithms are used to extract sentiment information from the text. This invention can employ a hybrid sentiment analysis method combining a sentiment dictionary and machine learning to accurately identify positive and negative sentiment words in the text. Alternatively, pre-trained sentiment analysis models or algorithms known in the field can also be used to accurately identify positive and negative sentiment words in the text, such as BERT (Bidirectional Encoder Representations from Transformers) and RoBERTa (Robustly optimized BERT pretraining approach).

[0063] Finally, the discrete sentiment lexical statistics are transformed into continuous, standardized sentiment scores using an improved Sigmoid activation function, as shown in the following formula:

[0064] ......(2)

[0065] in, Indicates the sentiment score; This represents the weighted count of positive sentiment words in the text. This represents the weighted count of negative sentiment words in the text. and These are adjustment parameters for positive and negative emotions, used to balance the influence weight of different emotion types on the final score. Their values ​​can be preset using empirical values ​​and later modified according to established rules to adapt to the emotional expression characteristics of different business scenarios. For example, the established rules could be: pre-configured for each business domain. and The value will be adjusted to reflect changes in the current business domain later on. and The value of is determined by . Furthermore, the established rules can also be dynamically adjusted based on the sensitivity of historical user feedback, thereby ensuring the accuracy and robustness of the sentiment quantification model. This mechanism can transform complex natural language feedback into quantifiable numerical signals, providing standardized input for subsequent steps.

[0066] (2) Analyze the implicit feedback behavior represented by the implicit feedback behavior data, determine whether the implicit feedback behavior is an implicit negative feedback behavior, and generate a behavior signal based on the determination result; wherein, when the implicit feedback behavior is an implicit negative feedback behavior, the behavior signal is 1, otherwise the behavior signal is 0.

[0067] This example uses implicit feedback behavior, specifically whether a user performs consecutive queries, as an example. The behavior signal can be implemented using the following binary judgment function:

[0068] ......(3)

[0069] in, Indicates behavioral signals; Indicates the first The query results of the current user query (i.e., this time, corresponding to the current question and answer pair); Indicates the first The query results for the secondary user; Indicates the first Secondary user query and the first The semantic similarity of query results between secondary user queries (cosine similarity or semantic similarity calculation methods based on pre-trained language models can be used). This represents the preset similarity threshold parameter; Indicates the first Secondary user query and the first The time interval between user queries; This represents the preset time threshold parameter.

[0070] These thresholds can be empirically determined through statistical analysis of historical user query logs to achieve the optimal balance between recall and precision. When both high similarity and short time intervals are met, the current implicit feedback behavior is determined to be implicit negative feedback behavior, and the output signal value is 1; otherwise, the output signal value is 0.

[0071] In addition, behavioral signals can also be implemented using mathematical models or machine learning classifiers.

[0072] (3) Normalize the quantitative scores, emotional scores and behavioral signals.

[0073] Specifically, the normalization formula is as follows:

[0074] ......(4)

[0075] in, This represents the normalized signal value of the target signal; the target signal can be a quantitative score, sentiment score, or behavioral signal. Represents the original signal value of the target signal; and These represent the minimum and maximum values ​​in the quantitative score, sentiment score, and behavioral signal, respectively.

[0076] Through this normalization process, the system can eliminate the dimensional differences between different signal types, ensuring the fairness and accuracy of the subsequent fusion process.

[0077] (4) The normalized quantitative score, emotional score and behavioral signal are weighted and calculated to obtain the comprehensive feedback score.

[0078] Specifically, the comprehensive feedback score can be calculated using the following formula:

[0079] ......(5)

[0080] in, This represents the overall feedback score; and Let represent the weighting coefficients of the quantitative score and the sentiment score, respectively, and satisfy . ; As an independent penalty coefficient, it is used to adjust the intensity of the deduction of the comprehensive feedback score by implicit negative feedback behavior; This represents the signal value after normalization of the quantified score. This represents the signal value after normalization of the sentiment score; This represents the signal value after normalization of the behavioral signal.

[0081] The above scheme uses a linear weighted calculation method to obtain the comprehensive feedback score. Alternatively, it can be replaced with a dynamic weighted model based on an attention mechanism or a small multilayer perceptron (MLP) network. This model can automatically learn and assign optimal fusion weights based on the intrinsic characteristics and interrelationships of different feedback signals, thus generating a more intelligent and adaptive comprehensive feedback score than a fixed-weight linear fusion method.

[0082] In this embodiment of the invention, the positive feedback category is specifically divided into a positive feedback category without natural language comments and a positive feedback category with natural language comments.

[0083] Negative feedback is specifically divided into two categories: negative feedback without natural language comments and negative feedback with natural language comments.

[0084] In this embodiment of the invention, the feedback category of the question-and-answer pair is determined based on the comprehensive feedback score and user feedback data, which may specifically include:

[0085] (1) When the overall feedback score is greater than or equal to the preset score threshold, and the user feedback data does not contain natural language comments, the feedback category is determined to be positive feedback and no natural language comments. The preset score threshold can be equal to 3.

[0086] (2) When the overall feedback score is greater than or equal to the preset score threshold and the user feedback data contains natural language comments, the feedback category is determined to be positive feedback with natural language comments.

[0087] (3) When the overall feedback score is less than the preset score threshold and the user feedback data contains natural language comments, the feedback category is determined to be negative feedback and has natural language comments.

[0088] (4) When the overall feedback score is less than the preset score threshold and the user feedback data does not contain natural language comments, the feedback category is determined to be negative feedback and no natural language comments.

[0089] Here, the comprehensive feedback score is calculated using multi-source user feedback data. Only when the comprehensive feedback score is greater than or equal to a preset score threshold will the feedback category be determined as a positive feedback category. This helps to ensure the data quality of the positive knowledge set.

[0090] In a specific example, a rule-based logic engine is used to determine the feedback category. Specifically, firstly, it is determined whether the overall feedback score is greater than or equal to a preset score threshold to obtain the first judgment result.

[0091] If the first judgment result is yes, then it is determined whether the user feedback data contains natural language comments. If the user feedback data does not contain natural language comments, then the feedback category is determined to be positive feedback and there is no natural language comment category. If the user feedback data contains natural language comments, then the feedback category is determined to be positive feedback and there is a natural language comment category.

[0092] If the first judgment result is negative, then it is determined whether the user feedback data contains natural language comments. If the user feedback data contains natural language comments, the feedback category is determined to be negative feedback with natural language comments. If the user feedback data does not contain natural language comments, it is determined whether there is implicit negative feedback behavior. If there is implicit negative feedback behavior, the feedback category is determined to be first negative feedback with no natural language comments. If there is no implicit negative feedback behavior, the feedback category is determined to be second negative feedback with no natural language comments. The negative feedback with no natural language comments category includes both the first negative feedback with no natural language comments category and the second negative feedback with no natural language comments category.

[0093] By meticulously categorizing feedback as described above, this application can more accurately reflect the user's true intent, thereby providing precise scenario guidance for subsequent knowledge base operations.

[0094] In addition, machine learning classification models, such as decision trees, support vector machines, or gradient boosting machines, can be used to determine the feedback category. By supervising the training of these models using historical feedback data, an intelligent classifier that can automatically learn and optimize its classification logic can be built, thereby improving the accuracy and generalization ability of scene recognition.

[0095] In this embodiment of the invention, when the feedback category is positive feedback and there is a natural language comment category, the target knowledge base operation strategy may include:

[0096] (1) Using a pre-defined language model, the answers in the question-and-answer pair are enhanced based on natural language comments to generate an enhanced version of the question-and-answer pair.

[0097] (2) Add the enhanced version of the question-and-answer pair to the positive knowledge set.

[0098] Specifically, positive feedback with natural language comments indicates that the user approves of the current query results and, based on this, offers supplementary suggestions or improvements. In this scenario, firstly, a pre-defined language model is invoked, inputting the original user question, the original answer, and the natural language comments. The language model includes a first prompt template to guide it in generating a new answer that includes the original answer and the improved natural language comments. In this invention, the pre-defined language model can be an LLM (Limited Language Model). Furthermore, the pre-defined language model can be replaced with a multi-agent collaborative system composed of multiple lightweight, expert-level language models dedicated to specific tasks. For example, expert models can be trained separately for different tasks such as fact-checking, text polishing, knowledge summarization, and content supplementation. A routing model distributes editing requests to the most suitable expert model or model combination for processing, thereby improving the efficiency and quality of knowledge editing.

[0099] Then, the original user question and the new answer generated by the language model are combined to form a new question-answer pair, an enhanced version of the question-answer pair, which is then added to the positive knowledge set. This achieves incremental optimization and improvement of knowledge. This mechanism not only preserves the basic content recognized by users but also enables continuous enhancement of knowledge through user wisdom.

[0100] In addition, relevant metadata can be added to the enhanced version of the question-answer pair, and this metadata will be added to the positive knowledge set along with the enhanced version of the question-answer pair. This metadata includes, for example, quantitative scores, natural language comments, feedback time, and quality tags.

[0101] The relevant storage data for question-and-answer pairs (such as new question-and-answer pairs and related metadata) can be stored in JSON format. In a specific example, the JSON format storage data for positive feedback with natural language comments would look like this:

[0102] json

[0103] {

[0104] "q": "Original user question, such as: How does the RAG system handle multimodal data?"

[0105] "a": "A new answer generated by the language model. This answer incorporates user feedback and provides a more comprehensive and detailed solution based on the original answer."

[0106] "metadata": {

[0107] "user_rating (user rating, i.e., quantitative rating)": 4.8,

[0108] "user_comment (user comments, i.e., natural language reviews)": "The answer is good; it is recommended to supplement with the specific processing flow of multimodal data."

[0109] "feedback_time": "2024-01-15T14:20:00Z",

[0110] "quality_label": "enhanced_quality",

[0111] "source_type": "llm_enhanced",

[0112] "original_answer": "RAG systems can process multimodal data using vectorization techniques..."

[0113] "llm_model (Large Language Model)": "gpt-4",

[0114] "enhancement_type": "content_supplement"

[0115] }

[0116] }

[0117] ```

[0118] In this embodiment of the invention, when the feedback category is negative feedback and there is a natural language comment category, the target knowledge base operation strategy may include:

[0119] (1) Set the confidence weight of the question-answer pair to the first preset weight. The value range of the first preset weight can be (0.8-1.0).

[0120] (2) Using a pre-defined language model, analyze natural language comments and question-answer pairs to generate an error analysis report; the enhanced version of the question-answer pairs includes the question-answer pairs and their error analysis reports and confidence weights.

[0121] (3) Add the enhanced version of the question-and-answer pair to the negative knowledge set.

[0122] Specifically, negative feedback with natural language comments indicates that the user does not agree with the current query results, and provides a specific reason for the error. In this scenario, firstly, the confidence weight of the question-answer pair is set to a first preset weight. Then, a preset language model is invoked, inputting the original user question, original answer, and natural language comments into the language model. The language model includes a second prompt template to guide the generation of an error analysis report, which contains the reason for the error in the original answer, the error category, and correction suggestions. The enhanced version of the question-answer pair includes the question-answer pair, its error analysis report, and confidence weight. Finally, the enhanced version of the question-answer pair is added to the negative knowledge set. In this way, the error analysis report provides richer error feature information, providing a more accurate basis for subsequent error identification and avoidance. This enhanced negative knowledge set mechanism not only achieves effective isolation of erroneous content, but more importantly, improves the error identification and prevention capabilities of this application through the error analysis report.

[0123] Furthermore, in scenarios with negative feedback and natural language comments, relevant metadata can be added to the enhanced version of the question-answer pair. This metadata will be added to the positive knowledge set along with the enhanced version of the question-answer pair. In a specific example, the JSON format data stored in a scenario with negative feedback and natural language comments is as follows:

[0124] json

[0125] {

[0126] "q": "Original user question, for example: What are some optimization methods for the vector retrieval algorithm in the RAG system?"

[0127] "a": "Original answer".

[0128] "metadata": {

[0129] "user_rating": 1.0,

[0130] "user_comment": "The method for calculating vector similarity in the answer is incorrect; it is recommended to correct the algorithm description."

[0131] "feedback_time": "2024-01-15T18:30:00Z",

[0132] "quality_label": "enhanced_negative_sample (Enhanced Negative Sample)",

[0133] "source_type": "user_corrected (user correction)",

[0134] "feedback_type": "low_rating_with_comment (low rating with comment, i.e., negative feedback with natural language comments)",

[0135] "confidence_weight": 0.9,

[0136] "error_signal_strength": "high"

[0137] },

[0138] "error_analysis": {

[0139] "llm_analysis_report (LLM analysis report)": "Based on LLM analysis, the main errors in this answer are: 1) Inaccurate description of the vector similarity calculation formula; 2) Lack of differentiation between different distance measurement methods; 3) Incorrect description of the applicable scenarios for the optimization strategy. It is recommended to avoid these error patterns in subsequent answers."

[0140] "error_categories": ["technical_inaccuracy", "conceptual_confusion", "incomplete_explanation"],

[0141] "llm_model (Large Language Model)": "gpt-4",

[0142] "analysis_time": "2024-01-15T18:32:00Z",

[0143] "error_severity": "high"

[0144] }

[0145] }

[0146] ```

[0147] Furthermore, in scenarios with negative feedback and natural language comments, specific tags can be added to the original answer, and the corresponding natural language comments can be associated with it as metadata. Subsequently, staff can manually review and correct the original answer with the specified tags.

[0148] In this embodiment of the invention, when the feedback category is negative feedback and there is no natural language comment category, the target knowledge base operation strategy may include:

[0149] (1) Set the confidence weight of the question-answer pair to the second preset weight. The value range of the second preset weight can be (0.6-0.8).

[0150] (2) Add the question-answer pairs assigned with confidence weights to the negative knowledge set.

[0151] Specifically, negative feedback without natural language comments indicates that the user does not accept the current query results. In this scenario, firstly, the confidence weight of the question-answer pair is set to a second preset weight. Then, the question-answer pairs assigned confidence weights are added to the negative knowledge set.

[0152] In a specific example, the JSON format data for negative feedback without natural language comments is stored as follows:

[0153] json

[0154] {

[0155] "q": "Original user question, such as: How to improve the recall rate of the RAG system?"

[0156] "a": "The original answer, which may contain factual errors, logical flaws, or fail to meet user needs."

[0157] "metadata": {

[0158] "user_rating": 1.5,

[0159] "feedback_time": "2024-01-15T16:45:00Z",

[0160] "quality_label": "negative_sample (negative sample)",

[0161] "source_type": "user_rejected (user rejected)",

[0162] "feedback_type": "low_rating_no_comment (low rating with no comments, i.e., negative feedback and no natural language comments)",

[0163] "confidence_weight": 0.7,

[0164] "error_signal_strength": "medium (medium)"

[0165] }

[0166] }

[0167] ```

[0168] In this embodiment of the invention, when the feedback category is positive feedback and there is no natural language comment category, the target knowledge base operation strategy may include:

[0169] Add the question-and-answer pairs to the positive knowledge set.

[0170] In a specific example, the JSON format data stored in a scenario of positive feedback without natural language comments is as follows:

[0171] json

[0172] {

[0173] "q": "Original user question, such as: How to optimize the search performance of the RAG system?"

[0174] "a": "The original answer may include specific technical solutions, implementation steps, and relevant suggestions. This answer has been verified by users with high ratings, ensuring the accuracy and practicality of the content."

[0175] "metadata": {

[0176] "user_rating": 4.5,

[0177] "feedback_time": "2024-01-15T10:30:00Z",

[0178] "quality_label": "high_quality (high quality)",

[0179] "source_type": "user_validated (User verified)"

[0180] }

[0181] }

[0182] ```

[0183] In the above scheme, the positive knowledge collection mechanism not only achieves effective knowledge accumulation, but more importantly, it establishes a knowledge quality certification system based on user feedback, ensuring that the content entering the positive knowledge collection has been actually verified and recognized by users.

[0184] Furthermore, by processing natural language comments through language models, a highly automated optimization loop driven by ordinary users is achieved. The system can directly extract knowledge from users' corrective or supplementary opinions and complete the editing of the knowledge base without the intervention of domain experts, thereby significantly reducing the system's operating costs and dependence on specific human resources, and improving the system's versatility and scalability.

[0185] In one specific embodiment, the positive knowledge set can be a positive knowledge base, and the negative knowledge set can be an anti-knowledge base. The positive and anti-knowledge bases are logically or physically separated.

[0186] In another specific embodiment, the positive knowledge set and the negative knowledge set can be located in a unified knowledge base, in which the data in the negative knowledge set is marked with negative examples.

[0187] The knowledge base can be stored in plain text format. Alternatively, it can be stored as a structured knowledge graph. In this case, knowledge base operations will be transformed into adding, modifying, or labeling nodes and relations (triples) in the graph. For example, LLM-assisted editing will generate new entities or relations, while a negative example knowledge base might be represented by labeling certain paths or sub-paths in the graph as "error paths."

[0188] Based on a general inventive concept, the present invention also provides a knowledge base optimization system for enhanced retrieval generation. Figure 2 This is a schematic diagram of the structure of a knowledge base optimization system for enhanced retrieval provided in an embodiment of the present invention. For example... Figure 2 As shown, this system includes:

[0189] The feedback receiving module 21 is used to acquire user feedback data associated with the question-answer pairs generated by the search enhancement generation RAG system, and to calculate the user's comprehensive feedback score for the question-answer pairs using the user feedback data. The user feedback data includes at least quantitative scores, natural language comments, and implicit feedback behavior data. The implicit feedback behavior data is used to determine whether implicit feedback behavior exists. Implicit feedback behavior is non-user-initiated feedback behavior that reflects whether the user is satisfied with the current query results.

[0190] The feedback classification module 22 is used to determine the feedback category of the question-and-answer pair based on the comprehensive feedback score and user feedback data; the feedback categories are divided into positive feedback category and negative feedback category.

[0191] The knowledge base operation module 23 is used to select a target knowledge base operation strategy from a variety of preset knowledge base operation strategies based on the feedback category, and execute the target knowledge base operation strategy. Specifically, when the feedback category is positive, the target data structure is added to the positive knowledge set; when the feedback category is negative, the target data structure is added to the negative knowledge set. The target data structure is a question-answer pair or its enhanced version. The negative knowledge set is used to, when responding to a new user query, penalize the original retrieval score of the query result based on the semantic similarity between the new user query and each original user question in the negative knowledge set, and then filter the query results based on the penalized retrieval score.

[0192] In practical applications, users submit queries to the RAG system. The RAG system generates and returns query results to the user based on positive and negative knowledge sets. Users can provide feedback on the question-and-answer pairs. The feedback receiving module 21 receives user feedback data and generates a comprehensive feedback score. The feedback classification module 22 determines the feedback category of the question-and-answer pairs based on the comprehensive feedback score and user feedback data. The knowledge base operation module 23 executes the target knowledge base operation strategy corresponding to the feedback category. Specifically, when the user feedback data includes natural language comments, the knowledge base operation module 23 calls a preset language model via an API (Application Programming Interface) to process the natural language comments. The language model can then return a new answer or an error analysis report (see the aforementioned description).

[0193] Optionally, the feedback receiving module 21 includes a quantitative scoring receiving unit, a natural language comment receiving unit, and an implicit feedback behavior data receiving unit. These units employ a multi-channel parallel processing architecture to improve data processing efficiency. Specifically, the quantitative scoring receiving unit receives quantitative scores and uses them directly as a score for subsequent calculations. The natural language comment receiving unit performs sentiment analysis on the natural language comment when it is received (i.e., the user feedback data contains the natural language comment) to quantify it into a sentiment score; when no natural language comment is received, the sentiment score is set to 0. The implicit feedback behavior data receiving unit receives implicit feedback behavior data, analyzes the implicit feedback behavior represented by the implicit feedback behavior data, determines whether the implicit feedback behavior is an implicit negative feedback behavior, and generates a behavior signal based on the determination result; wherein, when the implicit feedback behavior is an implicit negative feedback behavior, the behavior signal is 1, otherwise, the behavior signal is 0.

[0194] The feedback receiving module 21 also includes a comprehensive feedback score calculation unit, used for:

[0195] (1) Normalize the quantitative score, the emotional score and the behavioral signal.

[0196] (2) The normalized quantitative score, the emotional score and the behavioral signal are weighted and calculated to obtain the comprehensive feedback score.

[0197] Optionally, the positive feedback category is specifically divided into a positive feedback category without natural language comments and a positive feedback category with natural language comments.

[0198] The negative feedback categories are specifically divided into two categories: negative feedback without natural language comments and negative feedback with natural language comments.

[0199] Feedback classification module 22 is specifically used for:

[0200] (1) When it is determined that the comprehensive feedback score is greater than or equal to the preset score threshold and the user feedback data does not contain the natural language comments, the feedback category is determined to be the positive feedback and no natural language comments category.

[0201] (2) When it is determined that the comprehensive feedback score is greater than or equal to the preset score threshold, and the user feedback data contains the natural language comments, the feedback category is determined to be the positive feedback and natural language comment category.

[0202] (3) When it is determined that the comprehensive feedback score is less than the preset score threshold and the user feedback data contains the natural language comments, the feedback category is determined to be the negative feedback and has a natural language comments category.

[0203] (4) When it is determined that the comprehensive feedback score is less than the preset score threshold and the user feedback data does not contain the natural language comments, the feedback category is determined to be the negative feedback and no natural language comments category.

[0204] Optionally, the knowledge base operation module 23 includes an operation router, a positive knowledge set operator, a negative knowledge set builder, and a language model auxiliary editor.

[0205] The operation router is used to select a target knowledge base operation strategy from a variety of preset knowledge base operation strategies based on the feedback category, and then sends operation instructions to the corresponding units in the positive knowledge set operator, negative knowledge set builder, and language model auxiliary editor according to the target knowledge base operation strategy. The positive knowledge set operator is used to process instances of positive feedback categories, and the negative knowledge set builder is used to process instances of negative feedback categories. The language model auxiliary editor is used to generate new answers or error analysis reports by calling preset language models.

[0206] By operating the router to achieve intelligent task distribution, different types of feedback are automatically routed to the corresponding specialized processing modules. This refined multi-strategy parallel execution architecture ensures that the system can handle multiple different types of feedback at the same time. Each scenario has a dedicated processing module, which significantly improves the system's processing efficiency, response speed and processing accuracy.

[0207] Specifically, when the feedback category is positive feedback and there is a natural language comment category, the positive knowledge set operator and the language model auxiliary editor are used together to:

[0208] (1) Using a preset language model, the answers in the question-answer pair are enhanced based on the natural language comments to generate an enhanced version of the question-answer pair.

[0209] (2) Add the enhanced version of the question-answer pair to the positive knowledge set.

[0210] When the feedback category is negative feedback and there is a natural language comment category, the negative knowledge set builder and the language model auxiliary editor work together to:

[0211] (1) Set the confidence weight of the question-answer pair to the first preset weight.

[0212] (2) Using a preset language model, analyze the natural language comments and the question-answer pair to generate an error analysis report; the enhanced version of the question-answer pair includes the question-answer pair, its error analysis report, and confidence weights.

[0213] (3) Add the enhanced version of the question-answer pair to the negative knowledge set.

[0214] When the feedback category is negative feedback and there is no natural language comment category, the negative knowledge set builder is used to:

[0215] (1) Set the confidence weight of the question-answer pair to the second preset weight.

[0216] (2) Add the question-answer pairs assigned with confidence weights to the negative knowledge set.

[0217] When the feedback category is positive feedback and there is no natural language comment category, the positive knowledge set operator is used to:

[0218] Add the question-and-answer pairs to the positive knowledge set.

[0219] Optionally, the positive knowledge set is a positive knowledge base.

[0220] The set of negative knowledge forms an anti-knowledge base.

[0221] The positive knowledge base and the negative knowledge base are logically or physically separated from each other.

[0222] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0223] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0224] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0225] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0226] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0227] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0228] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0229] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0230] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for optimizing a knowledge base generated with enhanced retrieval capabilities, characterized in that, include: Acquire user feedback data associated with question-answer pairs generated by the retrieval enhancement generation RAG system, and use the user feedback data to calculate a comprehensive feedback score for the user on the question-answer pairs; the user feedback data includes at least the quantitative score and the implicit feedback behavior data from quantitative ratings, natural language comments, and implicit feedback behavior data. Implicit feedback behavior refers to non-user-initiated feedback behaviors that reflect whether a user is satisfied with the current query results. Based on the comprehensive feedback score and the user feedback data, the feedback category of the question-and-answer pair is determined; the feedback category is divided into positive feedback category and negative feedback category; Based on the feedback category, a target knowledge base operation strategy is selected from a variety of preset knowledge base operation strategies, and the target knowledge base operation strategy is executed; wherein, when the feedback category is the positive feedback category, the target data structure is added to the positive knowledge set; when the feedback category is the negative feedback category, the target data structure is added to the negative knowledge set; the target data structure is the question-answer pair or its enhanced version; The negative knowledge set is used to perform a punitive adjustment on the original retrieval score of the query results of the new user query based on the semantic similarity between the new user query and each original user question in the negative knowledge set when responding to a new user query, and then filter the query results based on the punitively adjusted retrieval score. Using the user feedback data, a comprehensive feedback score for the user's response to the question-and-answer pair is calculated, specifically including: When the user feedback data includes the natural language comments, sentiment analysis is performed on the natural language comments to quantify them into a sentiment score; when the user feedback data does not include the natural language comments, the sentiment score is determined to be 0. The implicit feedback behavior represented by the implicit feedback behavior data is analyzed to determine whether the implicit feedback behavior is an implicit negative feedback behavior, and a behavior signal is generated based on the determination result; wherein, when the implicit feedback behavior is an implicit negative feedback behavior, the behavior signal is 1, otherwise the behavior signal is 0. The quantitative score, the emotional score, and the behavioral signal are normalized. The normalized quantitative score, the emotional score, and the behavioral signal are weighted and calculated to obtain the comprehensive feedback score. The positive feedback categories are specifically divided into positive feedback without natural language comments and positive feedback with natural language comments. The negative feedback categories are specifically divided into negative feedback without natural language comments and negative feedback with natural language comments. Negative feedback with natural language comments means that the user does not agree with the current query results and provides a specific reason for the error. Negative feedback without natural language comments means that the user does not agree with the current query results. When the feedback category is negative feedback and there is a natural language comment category, the target knowledge base operation strategy includes: The confidence weight of the question-answer pair is set to the first preset weight; Using a pre-defined language model, the natural language comments and the question-answer pairs are analyzed to generate an error analysis report. The language model includes a second prompt template to guide the generation of the error analysis report, which contains the error reason, error category, and correction suggestions of the original answer. The enhanced version of the question-answer pairs includes the question-answer pairs, their error analysis reports, and confidence weights. Add the enhanced version of the question-answer pair to the negative knowledge set; When the feedback category is negative feedback and there is no natural language comment category, the target knowledge base operation strategy includes: The confidence weight of the question-answer pair is set to the second preset weight; The question-answer pairs assigned confidence weights are added to the negative knowledge set.

2. The method for optimizing the knowledge base generated by retrieval enhancement according to claim 1, characterized in that, Based on the comprehensive feedback score and the user feedback data, the feedback category of the question-and-answer pair is determined, specifically including: When the overall feedback score is greater than or equal to a preset score threshold, and the user feedback data does not contain the natural language comments, the feedback category is determined to be the positive feedback and no natural language comments category. When the overall feedback score is greater than or equal to a preset score threshold, and the user feedback data contains the natural language comments, the feedback category is determined to be the positive feedback category with natural language comments. When the overall feedback score is less than a preset score threshold and the user feedback data contains the natural language comments, the feedback category is determined to be the negative feedback category with natural language comments. When the overall feedback score is less than a preset score threshold and the user feedback data does not contain the natural language comments, the feedback category is determined to be the negative feedback and no natural language comments category.

3. The method for optimizing the knowledge base generated by retrieval enhancement according to claim 1, characterized in that, When the feedback category is positive feedback and there is a natural language comment category, the target knowledge base operation strategy includes: Using a preset language model, the answers in the question-answer pair are augmented based on the natural language comments to generate an enhanced version of the question-answer pair; The enhanced version of the question-and-answer pair is added to the positive knowledge set.

4. The method for optimizing the knowledge base generated by retrieval enhancement according to claim 1, characterized in that, When the feedback category is positive feedback and there is no natural language comment category, the target knowledge base operation strategy includes: Add the question-and-answer pairs to the positive knowledge set.

5. The method for optimizing the knowledge base generated by retrieval enhancement according to claim 1, characterized in that, The set of positive knowledge constitutes a positive knowledge base; The set of negative knowledge forms an anti-knowledge base; The positive knowledge base and the negative knowledge base are logically or physically separated from each other.

6. A knowledge base optimization system for enhanced retrieval generation, characterized in that, The retrieval-enhanced knowledge base optimization system, applied to the retrieval-enhanced knowledge base optimization method as described in any one of claims 1 to 5, comprises: The feedback receiving module is used to acquire user feedback data associated with question-answer pairs generated by the retrieval enhancement generation (RAG) system, and to calculate a comprehensive feedback score for the user on the question-answer pairs using the user feedback data. The user feedback data includes at least the quantitative score and the implicit feedback behavior data from quantitative scores, natural language comments, and implicit feedback behavior data. Implicit feedback behavior refers to non-user-initiated feedback behavior that reflects whether the user is satisfied with the current query results. The feedback classification module is used to determine the feedback category of the question-and-answer pair based on the comprehensive feedback score and the user feedback data; the feedback category is divided into positive feedback category and negative feedback category; The knowledge base operation module is used to select a target knowledge base operation strategy from a variety of preset knowledge base operation strategies based on the feedback category, and execute the target knowledge base operation strategy; wherein, when the feedback category is the positive feedback category, the target data structure is added to the positive knowledge set; when the feedback category is the negative feedback category, the target data structure is added to the negative knowledge set; the target data structure is the question-answer pair or its enhanced version; wherein, when responding to a new user query, the negative knowledge set is used to penalize the original retrieval score of the query result of the new user query based on the semantic similarity between the new user query and each original user question in the negative knowledge set, and then filter the query result based on the penalized retrieval score.

Images