FAQ knowledge base closed-loop optimization method based on million-level user message data
By combining embedded models and large language models, user comments are filtered and clustered to generate standardized FAQ question-answer pairs, which solves the problem of insufficient coverage of the FAQ knowledge base in high-traffic service scenarios and realizes efficient and automated optimization of the FAQ knowledge base.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- FUDAN UNIVERSITY
- Filing Date
- 2026-04-16
- Publication Date
- 2026-05-19
AI Technical Summary
In high-traffic service scenarios, existing technologies struggle to efficiently mine complex questions from millions of user comments and transform them into standardized FAQ pairs. Furthermore, the lack of a closed-loop verification mechanism optimized by the knowledge base results in insufficient coverage and matching accuracy.
An embedding model is used for vector similarity matching, and a large language model is used for semantic consistency verification. Difficult messages are filtered out. The large language model is used to extract core requests and intelligently cluster them to generate standardized FAQ question-answer pairs. The embedding model and the reordering model are used for answer matching and ranking. The large language model is used for verification to build a closed-loop optimization process.
It enables efficient identification and automated evaluation of difficult comments, improves the comprehensiveness and accuracy of the FAQ knowledge base, saves labor costs, and enhances the efficiency and accuracy of knowledge base updates.
Smart Images
Figure CN122064792A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of knowledge base technology, specifically to a closed-loop optimization method for FAQ knowledge bases based on millions of user comment data. Background Technology
[0002] FAQs (Frequently Asked Questions) typically refer to a knowledge base consisting of pre-written and organized "standardized" question-and-answer pairs, similar to a fixed reference manual.
[0003] Currently, there are two main types of FAQ optimization methods. One type is to generate new FAQ question-answer pairs based on product documents using RAG (Retrieval Enhancement Generation) technology (such as CN 119884335 A / CN 111046152 B); the other type is to generate extended synonyms based on the original FAQ question-answer pairs (such as CN 120336457 A): based on the existing FAQ question-answer pairs, expand the questions with synonymous expressions to enrich the matching dimensions of the question-answer pairs.
[0004] High-traffic service scenarios, such as government Q&A, often involve millions of user inquiries. In these scenarios, user requests cover a wide range of business details and vary greatly in expression, requiring higher levels of completeness and accuracy in the coverage and matching of FAQ pairs. However, existing technologies mainly focus on optimizing the FAQ pairs themselves, with few reports on optimization methods for FAQ pairs based on users' actual needs. In particular, there are technological gaps in the three stages of "mining difficult messages - FAQ generation - optimization effect verification".
[0005] Difficult comments cannot be automatically mined: It is impossible to efficiently extract difficult comments that "cannot be answered by the existing knowledge base" from millions of user comments by relying solely on manual methods; Transforming complex comments into FAQs: Current technology lacks a computerized conversion technology for "fragmented complex comments → standardized FAQs", making it impossible to summarize massive amounts of fragmented comments into structured question-and-answer pairs, and also unable to adapt to the diverse expressions of users; Lack of a closed-loop verification mechanism for knowledge base optimization: Existing technologies have not established a closed-loop process of "adding new question-answer pairs to the database - verifying the effect - adjusting parameters", and cannot quantify the improvement in coverage after knowledge base expansion. Summary of the Invention
[0006] To address the shortcomings of existing technologies, this invention provides a closed-loop optimization method for FAQ knowledge base based on millions of user comments, comprising the following steps; Step S1, Filtering of Difficult Messages: Based on the embedding model, the vector similarity of user messages and the original FAQ knowledge base is matched, and then the semantic consistency is verified by combining the large language model to filter out difficult messages that cannot be covered by the original FAQ knowledge base from the user messages. Step S2, High-Frequency FAQ Generation: For the difficult messages, the core needs are extracted and the questions are rewritten using a large language model to obtain the core questions; the core questions are intelligently clustered based on semantic similarity to generate standardized FAQ question-answer pairs; Step S3, Validation of Addition to Knowledge Base: Add the generated FAQ question-answer pairs to the knowledge base to form an optimized FAQ knowledge base; use the embedding model and the rearrangement model to match and sort the answers to the difficult messages, then use the large language model to judge the accuracy of the matching answers and calculate the effect evaluation index. If it is determined to be valid, proceed to the next step; otherwise, modify the parameters and return to step S1. Step S4: Knowledge base update: Add the verified FAQ question-answer pairs to the FAQ knowledge base.
[0007] In some embodiments, step S1 further includes; S101: Configure the data processing environment and large language model parameters; S102: Read user comment data and compare it with the original FAQ knowledge base question and answer pairs, and perform preprocessing; S103: Use an embedding model to vectorize text data and build a vector index library; S104: Use the vector index library to perform similarity retrieval on each user comment to obtain preliminary matching candidate questions; S105: Call the large language model to determine whether the user's comment can be answered by the candidate questions; S106: Output the filtered problematic message data.
[0008] In some embodiments, step S2 further includes; S201: Configure the data processing environment and large language model; S202: Read the problematic message data and preprocess it; S203: For each message: rewrite it into a core question using a large language model; vectorize the core question; retrieve similar representative questions; call the large language model to determine semantic similarity to determine its cluster affiliation or label it as a new question; S204: For each cluster, generate representative questions and standardized FAQ question-answer pairs using a large language model; output the clustering results.
[0009] In some embodiments, step S3 further includes; S301: Configure the data processing environment and model parameters; S302: Read the problematic message data and the optimized FAQ knowledge base question and answer pairs; S303: Construct the vector index of the optimized knowledge base; S304: Use an embedding model to perform a preliminary search on the difficult comments, then use a reordering model to reorder the results, and select the question-answer pair with the highest score as the simulated answer; S305: Call the large language model to determine whether the simulated answer can respond to the corresponding user message; S306: Calculate the knowledge base coverage improvement ratio or retrieval performance index based on the judgment results.
[0010] Beneficial effects This invention provides a closed-loop optimization method for FAQ knowledge base based on millions of user comments. Compared with existing technologies, it has the following advantages: This technology innovatively implements a closed-loop optimization system for FAQs based on millions of user comments. It constructs a two-stage retrieval mechanism of "embedded model vector similarity matching + large language model semantic consistency verification," which can efficiently identify difficult comment data and achieve high-precision semantic clustering. This enables accurate entry of high-frequency, difficult questions into the database and automated evaluation of the database addition results, significantly reducing the manual cost of expanding the knowledge base and improving the comprehensiveness and accuracy of the knowledge base coverage. Attached Figure Description
[0011] Figure 1 This is a schematic diagram of the process of the present invention. Figure 1 ; Figure 2 This is a schematic diagram of the process of the present invention. Figure 2 ; Figure 3 This is a schematic diagram of the process of the present invention. Figure 3 ; Figure 4 This is a schematic diagram of the process of the present invention. Figure 4 ; Figure 5 This is a schematic diagram of the system structure of the present invention. Detailed Implementation
[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0013] Please see Figure 1-5This invention provides a technical solution: a closed-loop optimization method for FAQ knowledge base based on millions of user comments, comprising the following steps; Step S1, Filtering of Difficult Messages: Based on the embedding model, the vector similarity of user messages and the original FAQ knowledge base is matched, and then the semantic consistency is verified by combining the large language model. Difficult messages that cannot be covered by the original FAQ knowledge base are filtered out from the user messages. By analyzing historical message data, message questions that cannot be answered by the question-answer pair in the current FAQ knowledge base are filtered out. Step S2, High-Frequency FAQ Generation: For the difficult messages, the core requirements are extracted and the questions are rewritten using a large language model to obtain core questions; based on semantic similarity, the core questions are intelligently clustered to generate standardized FAQ question-answer pairs; for the selected difficult messages, common questions are identified through clustering algorithms and corresponding FAQ question-answer pairs are generated to improve the coverage of the knowledge base. Step S3, Validation of Knowledge Base Addition: Add the generated FAQ question-answer pairs to the knowledge base to form an optimized FAQ knowledge base; use the embedding model and the rearrangement model to match and sort the answers to the difficult comments, then use the large language model to judge the accuracy of the matched answers and calculate the effect evaluation index. If it is determined to be valid, proceed to the next step; otherwise, modify the parameters and return to step S1 to verify the validity of the generated FAQ question-answer pairs, ensuring that these newly added question-answer pairs have high accuracy and practicality and meet user needs. Step S4, Knowledge Base Update: Add verified FAQ question-answer pairs to the FAQ knowledge base. Verified high-frequency FAQ question-answer pairs will be added to the knowledge base, and the existing FAQ content will be updated and optimized to provide a more comprehensive question and answer service.
[0014] In this embodiment, step S1 further includes; S101: Configure the data processing environment and large language model parameters; S102: Read user comment data and compare it with the original FAQ knowledge base question and answer pairs, and perform preprocessing; S103: Use an embedding model to vectorize text data and build a vector index library; S104: Use the vector index library to perform similarity retrieval on each user comment to obtain preliminary matching candidate questions; S105: Call the large language model to determine whether the user's comment can be answered by the candidate questions; S106: Output the filtered problematic message data; In the above steps, necessary libraries such as pandas, tqdm, httpx, OpenAI, and ThreadPoolExecutor are loaded; Secondly, configure the API client for the large language model (such as Xinference) and set the model name, hyperparameters, etc.
[0015] Furthermore, the user message data to be processed is read from the Excel file and preprocessed (such as removing blank values and format conversion).
[0016] Furthermore, it reads question-and-answer pairs from the original FAQ knowledge base from an Excel file.
[0017] Furthermore, user comment data and question-and-answer pairs from the FAQ knowledge base are processed and converted into Document objects to prepare for subsequent vectorization and retrieval.
[0018] Furthermore, an embedding model is used to convert document objects into vectors, and an index is built using FAISS (or other vector retrieval libraries) to support subsequent similarity retrieval.
[0019] Furthermore, tasks can be submitted using multiple threads; Each user comment is processed in parallel using ThreadPoolExecutor, and matched based on the similarity between the comment and questions in the knowledge base.
[0020] Furthermore, for each comment, FAISS is used to search for similar questions, and the most relevant standard questions are selected based on the returned similarity scores for preliminary matching.
[0021] Furthermore, a request is sent to the large language model to determine whether the question can be matched with existing candidate questions, and the standard question most relevant to the current question or False (indicating that there is no standard question among the candidate questions that can be used to answer the question).
[0022] Finally, the results of the large model's judgment (whether there are similar problems, the ranking of selected problems, etc.) are recorded in a DataFrame and saved to a specified output file (such as Excel).
[0023] In this embodiment, step S2 further includes; S201: Configure the data processing environment and large language model; S202: Read the problematic message data and preprocess it; S203: For each message: rewrite it into a core question using a large language model; vectorize the core question; retrieve similar representative questions; call the large language model to determine semantic similarity to determine its cluster affiliation or label it as a new question; S204: For each cluster, generate representative questions and standardized FAQ question-answer pairs using a large language model; output the clustering results.
[0024] In the steps above, necessary Python libraries are loaded, such as pandas, tqdm, httpx, and OpenAI. Simultaneously, the model used for querying, such as a large language model (e.g., Qwen2.5), is initialized, and relevant hyperparameters are set.
[0025] Furthermore, the problematic message data obtained in step D is read from the specified Excel file, and data preprocessing is performed, such as removing blank values and converting data formats, to prepare for subsequent calculations.
[0026] Furthermore, each user comment is submitted to a task processing queue using multi-threaded processing. The purpose of the task is to process the comment data, ensuring efficient computation and querying.
[0027] Furthermore, the core demands of user comments are extracted and rewritten into semantically similar core questions.
[0028] Furthermore, the core issue is vectorized, converting each message into a vector representation. The purpose of this step is to transform the text data into a form that computers can understand and use for similarity comparison.
[0029] Furthermore, a pre-trained model (such as Bge-m3) is used to perform similarity retrieval on the transformed comments (the core question), finding the most matching candidate questions and returning the retrieval results. The goal of the retrieval is to identify the Top-k representative questions most relevant to the current comment.
[0030] Furthermore, the top-k candidate representative questions are traversed to determine whether the current message belongs to a specific representative question. By invoking the large language model, it is determined whether the current candidate representative question belongs to the same category as the core question in the user's message. The criterion for this determination is whether a core question can be matched and whether it meets the business requirements.
[0031] Furthermore, if the current core problem matches a suitable candidate representative problem, the problem is marked as the cluster to which the corresponding representative problem belongs.
[0032] Furthermore, if no suitable candidate question is found for the current core question, it is considered a new question. In this case, the question is added to the candidate pool for further processing. This step ensures that new questions are not overlooked and allows for continuous optimization of the FAQ knowledge base.
[0033] Specific Implementation Example 10: A41: Group the clusters corresponding to each representative problem, randomly select a certain number of core problems from the clusters, and use a large model to extract and summarize common problems as new representative problems of the clusters.
[0034] Furthermore, all processed results (each core problem, corresponding cluster ID, and representative problem of the cluster) are summarized into a DataFrame and saved to a specified output file, such as an Excel file; In this embodiment, step S3 further includes; S301: Configure the data processing environment and model parameters; S302: Read the problematic message data and the optimized FAQ knowledge base question and answer pairs; S303: Construct the vector index of the optimized knowledge base; S304: Use an embedding model to perform a preliminary search on the difficult comments, then use a reordering model to reorder the results, and select the question-answer pair with the highest score as the simulated answer; S305: Call the large language model to determine whether the simulated answer can respond to the corresponding user message; S306: Calculate the knowledge base coverage improvement ratio or retrieval performance index based on the judgment results.
[0035] In the above steps, necessary libraries such as pandas, tqdm, httpx, OpenAI, and ThreadPoolExecutor are loaded; Additionally, configure the API client for the large language model (such as Xinference) and set the model name, hyperparameters, etc.
[0036] Furthermore, the system reads the problematic message data from the Excel file and performs data preprocessing (such as removing blank values and format conversion).
[0037] Furthermore, it reads question-and-answer pairs from the optimized FAQ knowledge base in an Excel file.
[0038] Furthermore, user comment data and question-and-answer pairs from the FAQ knowledge base are processed and converted into Document objects to prepare for subsequent vectorization and retrieval.
[0039] Furthermore, an embedding model is used to convert document objects into vectors, and an index is built using FAISS (or other vector retrieval libraries) to support subsequent similarity retrieval.
[0040] Furthermore, tasks can be submitted using multiple threads; Each user comment is processed in parallel using ThreadPoolExecutor, and matched based on the similarity between the comment and questions in the knowledge base.
[0041] Furthermore, for each comment, FAISS is used to search for similar questions, and the most relevant standard questions are selected based on the returned similarity scores for preliminary matching.
[0042] Furthermore, a re-ranking model (such as bge-reranker) can be used to sort similar questions, and the first one can be taken as the final answer.
[0043] We simulated the FAQ system's responses to user inquiries using an "embedding model + rearrangement model," instead of using the original FAQ system's framework. The aim was to make the optimization solution more adaptable and portable. By testing in an independent environment, we can ensure the universality of the optimization workflow, rather than being limited to a specific FAQ framework. Later, by replacing the retrieval and sorting modules in the simulation environment, we can independently evaluate the workflow's performance on different systems without relying on a specific framework. In this way, we can optimize for the characteristics of different FAQ systems, ensuring stable and effective operation in various real-world application scenarios.
[0044] Furthermore, a request is sent to the large language model to determine whether the user's comment can be answered by the first question-and-answer pair selected by the reordering model.
[0045] Furthermore, calculate the percentage of difficult comments that can be answered after this database addition. Alternatively, use data that has been reviewed and annotated by professionals as a test set to calculate the NDCG and MRR metrics for the retrieval task, serving as a reference for parameter adjustment.
[0046] The system is based on a collaborative computing architecture of embedded models and large language models, and has built an end-to-end intelligent FAQ optimization and processing framework, which mainly includes four functional modules: multi-source data access module, difficult question screening module, high-frequency question semantic clustering module, and database addition effect verification module.
[0047] This technology innovatively employs a collaborative mechanism of "embedded model vector similarity matching + large language model semantic consistency verification + text clustering adapted to millions of data points" to achieve dynamic and accurate upgrades of the FAQ knowledge base. This closed-loop optimization technique saves processing time for large-scale data and improves the usability of newly added question-answer pairs. This technology is original in the field of large-scale FAQ knowledge base optimization.
[0048] The core technical points and protection points of this invention include the following aspects: (1) A two-stage retrieval mechanism of “embedded model vector similarity matching + large language model semantic consistency verification” This invention innovatively combines vector matching technology from embedded models with the validation capabilities of large language models to build an intelligent system. This system can efficiently filter unanswered questions from users' historical comments that are not currently covered by the FAQ knowledge base. Through this mechanism, the system can automatically analyze and determine whether relevant question-answer pairs can be found in the existing knowledge base, thereby identifying knowledge domains that need to be expanded.
[0049] (2) Large-scale intelligent text clustering algorithm based on semantic similarity This invention proposes an intelligent clustering algorithm based on semantic similarity. Utilizing few-shot learning and carefully designed prompts, it helps large language models determine the similarity between questions. This algorithm can automatically summarize a large number of scattered user comments into standardized FAQ question-answer pairs. This clustering algorithm requires no manual intervention, greatly improving the system's adaptability, accuracy, and automation level, making it particularly suitable for processing massive amounts of data.
[0050] A closed-loop verification mechanism for rapid knowledge base updates and intelligent evaluation This invention establishes a closed-loop verification mechanism by combining traditional similarity retrieval with intelligent judgment using a large language model. After adding a new FAQ question-answer pair, this mechanism quickly matches related questions using vector similarity and selects the final answer using a reordering model. Finally, a large language model is used to intelligently verify the matched standard questions, ensuring the accuracy and high quality of the new question-answer pairs. This technology reduces the workload of manual review, improves the efficiency of knowledge base updates, and ensures that the system can maintain high efficiency and accuracy while rapidly expanding.
[0051] This invention utilizes embedded model and large language model technologies to build a knowledge base intelligent optimization system, completing the following three core workflows.
[0052] (1) By combining embedding model vector matching with semantic consistency verification of large language model, a system was built to filter out difficult comments that cannot be covered by the current FAQ knowledge base from users' historical comments. The system analyzes the historical comments entered by users to determine whether there are relevant question-answer pairs in the current FAQ database, thereby quickly identifying which difficult user comments need to be used as a reference for expanding the knowledge base.
[0053] (2) A smart clustering algorithm based on semantic similarity for processing large-scale data is proposed to find high-frequency questions. Through few-shot learning and Prompt engineering, a large language model is used to rewrite user comments (Qeury Rewriting) to extract core questions; and the large model is used to judge the semantic similarity between core questions, so as to automatically summarize massive scattered comments into standardized FAQ question-answer pairs, avoid manual intervention, and improve the adaptability and accuracy of the system.
[0054] (3) A closed-loop verification mechanism was established by combining traditional similarity retrieval with large language model judgment. After the newly added FAQ question-answer pairs are added to the database, potentially related questions are quickly matched based on vector similarity on the original difficult data. Then, a re-ranking model is used to select the question-answer pair with the highest score as the answer. The large language model performs intelligent verification on the matched question-answer pairs to ensure the accuracy of the final question-answer pairs. This mechanism effectively reduces the workload of manual review and ensures efficient processing of questions and the quality of the knowledge base.
[0055] Furthermore, any content not described in detail in this specification is existing technology known to those skilled in the art.
Claims
1. A closed-loop optimization method for FAQ knowledge base based on millions of user comments, characterized in that: Includes the following steps; Step S1, Filtering of Difficult Messages: Based on the embedding model, the vector similarity of user messages and the original FAQ knowledge base is matched, and then the semantic consistency is verified by combining the large language model to filter out difficult messages that cannot be covered by the original FAQ knowledge base from the user messages. Step S2, High-Frequency FAQ Generation: For the difficult messages, the core needs are extracted and the questions are rewritten using a large language model to obtain the core questions; the core questions are intelligently clustered based on semantic similarity to generate standardized FAQ question-answer pairs; Step S3, Validation of Addition to Knowledge Base: Add the generated FAQ question-answer pairs to the knowledge base to form an optimized FAQ knowledge base; use the embedding model and the rearrangement model to match and sort the answers to the difficult messages, then use the large language model to judge the accuracy of the matching answers and calculate the effect evaluation index. If it is determined to be valid, proceed to the next step; otherwise, modify the parameters and return to step S1. Step S4: Knowledge base update: Add the verified FAQ question-answer pairs to the FAQ knowledge base.
2. The closed-loop optimization method for FAQ knowledge base based on millions of user comments data as described in claim 1, characterized in that: Step S1 further includes: S101: Configure the data processing environment and large language model parameters; S102: Read user comment data and compare it with the original FAQ knowledge base question and answer pairs, and perform preprocessing; S103: Use an embedding model to vectorize text data and build a vector index library; S104: Use the vector index library to perform similarity retrieval on each user comment to obtain preliminary matching candidate questions; S105: Call the large language model to determine whether the user's comment can be answered by the candidate questions; S106: Output the data of the filtered problematic messages.
3. The closed-loop optimization method for FAQ knowledge base based on millions of user comments data as described in claim 2, characterized in that: Step S2 also includes: S201: Configure the data processing environment and large language model; S202: Read the data of the problematic message and preprocess it; S203: For each message: rewrite it into a core question using a large language model; vectorize the core question; Search for similar representative questions; Call the large language model to determine semantic similarity to identify its cluster or label it as a new problem; S204: For each cluster, use a large language model to generate representative questions and standardized FAQ question-answer pairs; Output the clustering results.
4. The closed-loop optimization method for FAQ knowledge base based on millions of user comments data as described in claim 3, characterized in that: Step S3 also includes: S301: Configure the data processing environment and model parameters; S302: Read the data of the problematic messages and the optimized FAQ knowledge base question-and-answer pairs; S303: Construct the vector index of the optimized knowledge base; S304: Use an embedding model to perform a preliminary search on the difficult comments, then use a reordering model to reorder the results, and select the question-answer pair with the highest score as the simulated answer; S305: Call the large language model to determine whether the simulated answer can respond to the corresponding user message; S306: Calculate the knowledge base coverage improvement ratio or retrieval performance index based on the judgment results.