Method for generating improved question-answer pair quality by using retrieval enhancement
By constructing high-quality question-and-answer pairs specific to the scenic area, using TF-IDF and K-Means clustering algorithms to select seed question-and-answer pairs, and combining them with retrieval enhancement generation technology, the problem of knowledge deficiency in large language models in cultural tourism scenic areas has been solved, achieving efficient and accurate question-and-answer services, and improving the tourist experience and the competitiveness of the scenic area.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TAIYUAN UNIVERSITY OF TECHNOLOGY
- Filing Date
- 2025-12-31
- Publication Date
- 2026-04-14
AI Technical Summary
Existing large language models lack specific knowledge for cultural tourism scenic spots, resulting in poor question-and-answer performance, an inability to respond to diverse tourist questions in a timely and effective manner, and an impact on the tourist experience.
By selecting a language model, acquiring and cleaning relevant corpora, generating question-answer pairs, selecting seed question-answer pairs, retrieval enhancement to generate reconstructed question-answer pairs, and manual annotation, high-quality question-answer pairs for specific scenic spots are constructed. TF-IDF and K-Means clustering algorithms are used to select seed question-answer pairs, and retrieval enhancement generation technology is combined to improve the quality of question-answer pairs.
It significantly improved the accuracy and diversity of question-and-answer pairs, shortened response time, enhanced the visitor experience, strengthened the scenic area's competitive advantage, reduced operating costs, and promoted the intelligent management of the scenic area and the development of smart tourism.
Smart Images
Figure CN121858702A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of natural language processing-annotated data acquisition and quality assessment in cultural and tourism industries, and more specifically, to a method for improving the quality of question-answer pairs by utilizing retrieval enhancement. Background Technology
[0002] In recent years, large language models have been widely used as a technological foundation for building AI tour guides for tourist attractions. However, in the field of cultural tourism, each scenic spot has unique introductions and historical stories, and commonly used large language models often lack this specific knowledge, resulting in unsatisfactory performance in actual scenic spot tours. The main factors causing this problem include: firstly, the accumulation of textual information for many specific scenic spots is relatively scarce; secondly, the problems in the cultural tourism field are unique, and existing technologies are insufficient to build effective question-answer pairs for these problems.
[0003] Limited staff within scenic areas make it difficult to respond quickly and effectively to the diverse questions raised by tourists. This often results in delayed responses from human staff, negatively impacting the tourist experience. To address the unique challenges of constructing question-answering pairs in the field of natural language processing within cultural tourism, particularly in themed scenic areas, there is an urgent need for intelligent question-answering pairs that can not only fill the knowledge gaps in existing large language models regarding specific scenic areas but also meet tourists' precise needs for information about the area. Summary of the Invention
[0004] To address the shortcomings of existing technologies, this invention aims to construct a comprehensive, high-quality question-and-answer pair set specifically for scenic areas. The entire process involves selecting a language model, acquiring and cleaning relevant corpora, generating question-and-answer pairs, selecting seed question-and-answer pairs, retrieval enhancement to reconstruct question-and-answer pairs, and manual annotation. Retrieval enhancement technology is employed to improve the quality of question-and-answer pairs using natural language models specific to a niche domain. This invention solves the thorny problems of information chaos, data redundancy, and scattered data provided by scenic areas to tourists. It represents a refined service offered by scenic areas to tourists, developing solutions based on tourist needs, effectively eliminating existing defects, responding promptly to tourist feedback, and addressing specific problems to improve the tourist experience and enhance the competitive advantage of scenic areas.
[0005] To achieve the above objectives, the present invention provides the following technical solution: A method for improving the quality of question-answer pairs based on retrieval-enhanced generation techniques is proposed. This method selects a language model for improving question-answer pair quality, cleans and reduces the duplication of materials from various channels and of different types, constructs a comprehensive text database of the objects involved in the question-answer pairs, generates original question-answer pairs using the language model, and then uses TF-IDF and K-Means clustering algorithms to jointly select the most representative and diverse seed question-answer pairs. The seed question-answer pairs are then reconstructed using retrieval-enhanced generation techniques, and annotation feedback is provided to improve the quality of the question-answer pairs. The specific steps include: Step 1. Select a language model and acquire and clean the text data: Select multiple different natural language model APIs for the question-answer pair generation task; use web crawling tools and field research to collect text data related to the objects involved in the question-answer pairs, remove irrelevant and repetitive content from the text data, and classify and organize the text data according to the semantic logical order to form a comprehensive text database of question-answer pairs. Step 2. Generate question-answer pairs: Use the natural language model to generate several core question-answer pairs that conform to semantic consistency based on the comprehensive text database obtained in Step 1; Step 3. Select seed question-answer pairs: Calculate the word frequency and importance of the core question-answer pairs obtained in Step 2 using the TF-IDF method, convert the word frequency and importance into feature vectors of the corresponding question-answer pairs, perform cluster analysis, and select the most representative and diverse seed question-answer pairs; Step 4. Retrieval Enhancement Generation and Reconstruction of Question-Answer Pairs: Using the comprehensive text database formed in Step 1 as a vector database, retrieval enhancement generation technology is used to reconstruct the seed question-answer pairs to ensure the accuracy and diversity of the question-answer pairs; Step 5. Manually annotate question-answer pairs: Build a network annotation system to provide the reconstructed seed question-answer pairs to humans for evaluation and collect human feedback in a timely manner.
[0006] Furthermore, in step 1, the specific steps for selecting a language model are as follows: Step 1.1: Construction of Standard Question Answering Dataset: Based on the domain and objectives of question answering pair construction, a standard question answering dataset was established as a benchmark test set, and multiple candidate language models were selected to answer questions based on the given standard question answering dataset; Step 1.2 Comprehensive evaluation and selection of language models: Use a comprehensive evaluation method to quantify all language models and select the language model used to improve the quality of question-answer pairs.
[0007] Furthermore, the comprehensive evaluation method scores the answers generated by other models using each model across several dimensions: accuracy, completeness, relevance, clarity, and politeness. It calculates the mean score, standard deviation, Kolmogorov-Smirnov (KS) statistic, and p-value for each language model at different levels.
[0008] Furthermore, in step 2, the core question-and-answer pair generation process is as follows: Step 2.1 Text Segmentation: The input text is segmented according to the newline character to form several independent paragraphs. The segmented independent paragraphs are used as input, and a question-answer generation task is performed using a preset language model to obtain the initial question-answer pairs corresponding to the content of each paragraph. Step 2.2 Dynamic Prompts: Calculate the corresponding number of question-and-answer generated N based on the character length L of each independent paragraph, where... C is the preset word count step constant; the paragraph content and the number of generated words N are filled into the preset prompt template to construct targeted dynamic prompts and ensure the high quality and relevance of the generated question-and-answer pairs; Step 2.3 Generate core question-answer pairs: Based on the paragraph length of the input text, the language model used to improve the quality of question-answer pairs generates a number of high-quality core question-answer pairs that match the paragraph length, and saves them as question-answer pair text.
[0009] Furthermore, in step 3, the process of selecting seed question-answer pairs is as follows: Step 3.1 Remove stop words and perform word segmentation: Filter the core question-answer pairs obtained in Step 2 for stop words, merge the questions and answers, and then perform word segmentation to improve the relevance of the search. Step 3.2 Feature Extraction and Vectorization: The TF-IDF method is applied to convert the word segmentation results into vectors. The term frequency (TF) of each word in a single question-answer pair and the inverse document frequency (IDF) in the entire question-answer pair set are calculated. The TF and IDF are multiplied to obtain the weight score of each word to measure the importance of the word. Based on the weight score, each question-answer pair is converted into a feature vector to form a TF-IDF sparse feature matrix. Step 3.3 Determine the optimal number of clusters K: Use the elbow rule and silhouette coefficient to determine the optimal number of clusters K to ensure the best clustering results; Step 3.4 T-SNE Dimensionality Reduction Weights and Cluster Density Calculation: The T-SNE method is used to reduce the dimensionality of high-dimensional features, and the optimal number of clusters K selected in Step 3.3 is used to execute the clustering algorithm to generate K different clusters; the density within each cluster is calculated to evaluate the clustering quality and distribution. Step 3.5 Resampling to generate seed question-answer pairs: Based on the coordinate distribution of each cluster in the reduced-dimensional space in Step 3.4, calculate the cluster density of each cluster, and resample the core question-answer pairs in Step 2 according to the cluster density to ensure data diversity. Finally, new question-answer pairs are generated and saved as question-answer pair text.
[0010] Furthermore, in step 4, the specific operations for retrieving and enhancing the generation of reconstructed question-answer pairs are as follows: Step 4.1 Establish a vector database: Perform text segmentation on the comprehensive text database from Step 1, and use the model provided by Hugging Face to create a vector database for fast retrieval and comparison; Step 4.2 Question Rewriting: After vectorizing the seed questions in the seed question-answer pair, retrieve documents related to the seed questions from the vector database. Improve the accuracy of the retrieval by rewriting the questions, and integrate the retrieval results of different questions. Sort the documents according to the semantic relevance between the rewritten seed questions and the retrieved documents and the information importance of the retrieved documents in the comprehensive text database in Step 1. Select the top M positions after sorting to ensure that the relevant corpus of the top M positions in the re-sorting sequence is covered as the technical boundary of the retrieval scope. Step 4.3 Call the language model to generate question-answer pairs for expert annotation: Use the documents related to the seed questions retrieved in Step 4.2 as reference materials. When calling the language model used to improve the quality of question-answer pairs to generate question-answer pairs to be annotated, ensure the accuracy of the question-answer pairs and improve the quality of the original question-answer pairs.
[0011] Furthermore, in step 5, the manually annotated question-and-answer pairs allow domain experts to annotate and evaluate the question-and-answer pairs obtained in step 4 through a set web interface. An evaluation mechanism is introduced based on the user feedback form to measure the quality of expert feedback and select the best feedback to improve the question-and-answer pairs to be annotated, thus forming the final question-and-answer pair library.
[0012] In summary, the invention has the following beneficial effects: This invention introduces a Large Language Model (LLM) for cleaning the raw dataset. Leveraging the powerful natural language processing capabilities of LLM, it efficiently identifies and removes noisy data, redundant information, and irrelevant text content, providing high-quality and effective text resources for subsequent data analysis and processing steps. This significantly improves the quality of the input data and ensures the accuracy and efficiency of subsequent processing steps.
[0013] To improve the quality of question-answer pairs, this invention applies Retrieval-Augmented Generation (RAG) technology, which combines the advantages of retrieval-based models and generative models. When answering questions, it not only relies on the knowledge of pre-trained language models, but also retrieves relevant information from external documents to supplement the answers, thereby ensuring the accuracy and timeliness of the answers and effectively improving the overall performance and service quality of the question-answering system.
[0014] To address the issue of question-and-answer pair accuracy, this invention first utilizes a large-scale model to automatically generate preliminary question-and-answer pairs. Then, a professional annotation team manually reviews and corrects these pairs. By deeply mining historical data within the scenic area and leveraging the professional knowledge of its staff, a comprehensive question-and-answer pair database is created, providing tourists with accurate information about specific scenic spots. This significantly shortens response time, improves information accuracy, and enhances the tourist experience. The combination of automation and human intervention leverages the large-scale model's ability to quickly generate a large number of candidate answers while also utilizing the experience and judgment of human experts to improve the accuracy of the final output question-and-answer pairs. This ensures that the information provided is both professional and easy for users to understand, greatly enhancing the quality of the question-and-answer pairs, effectively reducing manual labor requirements, lowering operating costs, and ultimately promoting the intelligent management of scenic areas and the development of smart tourism. Attached Figure Description
[0015] Figure 1 This is a flowchart illustrating the implementation process of the present invention.
[0016] Figure 2 A flowchart illustrating the process of generating reconstructed question-answer pairs to enhance retrieval.
[0017] Figure 3 A comparison chart showing the scores of the old and new answers.
[0018] Figure 4 A comparison chart of the score distribution between the old and new answers.
[0019] Figure 5 This is a comparison chart of the technical indicators of the old and new answers. Detailed Implementation
[0020] The present invention will now be described in further detail with reference to the accompanying drawings.
[0021] like Figures 1-5 As shown, this invention discloses a method for improving the quality of question-answer pairs based on retrieval enhancement generation technology. The method involves selecting a language model for improving question-answer pair quality, cleaning and deduplicating various types of data from multiple channels to construct a comprehensive text database of objects involved in the question-answer pairs, generating original question-answer pairs using the language model, and then using TF-IDF and K-Means clustering algorithms to jointly select the most representative and diverse seed question-answer pairs. The seed question-answer pairs are then reconstructed using retrieval enhancement generation technology, and annotation feedback is provided to improve the quality of the question-answer pairs. Specifically, the method includes the following steps: Step 1. Select a language model and acquire and clean the text data: Select multiple different natural language model APIs for the question-answer pair generation task; collect text data related to the objects involved in the question-answer pairs using web crawling tools and field research; remove irrelevant and repetitive content from the text data, and classify and organize the text data according to semantic logical order to form a comprehensive text database of question-answer pairs; the specific operation of selecting the language model is as follows: Step 1.1: Construction of Standard Question Answering Dataset: Based on the domain and objectives of question answering pair construction, a standard question answering dataset was established as a benchmark test set, and multiple candidate language models were selected to answer questions based on the given standard question answering dataset; Step 1.2 Comprehensive Evaluation and Selection of Language Models: A comprehensive evaluation method is used to quantify all language models and select the language model to improve the quality of question-answer pairs. To further quantify the performance of each model, this invention employs a comprehensive evaluation method. This method uses each model to score the answers generated by other models from the following dimensions: accuracy, completeness, relevance, clarity, and politeness. Simultaneously, the mean score, standard deviation, Kolmogorov-Smirnov (KS) statistic, and p-value are calculated for each model across different dimensions. The mean score measures the overall quality of the model's responses; the standard deviation reflects the consistency and reliability of the model's scoring results; and the KS statistic and p-value are used together to test whether the model's responses are significantly consistent with the preset standard answers. This method ensures the scientific rigor and fairness of the model selection process. After comprehensive evaluation, the language model selected to improve the quality of question-answer pairs is chosen.
[0022] The comprehensive evaluation method scores the answers generated by other models using each model across several dimensions: accuracy, completeness, relevance, clarity, and politeness. It calculates the mean score, standard deviation, Kolmogorov-Smirnov (KS) statistic, and p-value for each language model.
[0023] Step 2. Generate Question-Answer Pairs: Using a natural language model, generate several core question-answer pairs that conform to semantic consistency based on the comprehensive text database obtained in Step 1. Collect textual data related to the objects involved in the question-answer pairs using web crawling tools and field research. Clean the collected data, removing irrelevant and duplicate content to ensure the diversity and accuracy of data sources. Classify and organize the data according to semantic logical order. Finally, construct a comprehensive text database of the objects involved in the question-answer pairs. Use Selenium automated web crawling tools to crawl web text resources from various channels, including news reports, personal blogs, social media, academic literature, and official documents, related to the objects involved in the question-answer pairs, and save them as electronic text. Collect historical background information related to the objects involved in the question-answer pairs through field research, and organize and save it as electronic text. Filter non-Chinese characters using regular expressions and match Chinese text in the text. Use jieba word segmentation to divide the text into words, and apply TF-IDF (term frequency–inverse document frequency) technology to measure the importance of words to the text, identifying and removing irrelevant and duplicate information. To improve text relevance and consistency, a language model used to improve the quality of question-answer pairs is invoked to further process the cleaned and deduplicated text. Finally, the cleaned text is saved to build a comprehensive text database of the objects involved in the question-answer pairs.
[0024] The core question-and-answer pair generation process is as follows: Step 2.1 Text Segmentation: The input text is segmented based on line breaks to form several independent paragraphs. These independent paragraphs are then used as input to perform a question-and-answer generation task using a pre-defined language model, obtaining initial question-and-answer pairs corresponding to the content of each paragraph.
[0025] Step 2.2 Dynamic Prompts: Calculate the corresponding number of question-and-answer generated N based on the character length L of each independent paragraph, where... C is the preset word count step constant; the paragraph content and the number of generated words N are filled into the preset prompt template to construct targeted dynamic prompts and ensure the high quality and relevance of the generated question-and-answer pairs; High quality is defined by semantic consistency, meaning that only question-answer pairs that satisfy the criterion that "all facts in the answer can be found in the original paragraph" are considered high quality. Targetedness is defined by the strong binding between the prompt template and the paragraph length, meaning that the prompt explicitly constrains the model to extract information around the "current paragraph," and the number of generated prompts, N, is proportional to the paragraph's information entropy.
[0026] Step 2.3 Generate core question-answer pairs: Based on the paragraph length of the input text, the language model used to improve the quality of question-answer pairs generates a number of high-quality core question-answer pairs that match the paragraph length, and saves them as question-answer pair text. In actual operation, a constant C is set according to the paragraph length. The constant C is used as the length of a single group of text to divide the paragraph into several groups. For example, if the paragraph is 400 words long and the constant C is set to 200, then the number of groups for this paragraph is 2.
[0027] Step 3. Selecting Seed Question-Answer Pairs: Calculate the term frequency and importance of the core question-answer pairs obtained in Step 2 using the TF-IDF (Term Frequency-Inverse Document Frequency) method. Transform the term frequency and importance into feature vectors for the corresponding question-answer pairs, perform cluster analysis, and select the most representative and diverse seed question-answer pairs. The process for selecting seed question-answer pairs is as follows: Step 3.1 Remove stop words and perform word segmentation: Filter the core question-answer pairs obtained in Step 2 for stop words, merge the questions and answers, and then perform word segmentation to improve the relevance of the search.
[0028] Step 3.2 Feature Extraction and Vectorization: The TF-IDF method is applied to convert the word segmentation results into vectors. The term frequency (TF) of each word in a single question-answer pair and the inverse document frequency (IDF) in the entire question-answer pair set are calculated. The TF and IDF are multiplied to obtain the weight score of each word to measure its importance. Based on the weight score, each question-answer pair is converted into a feature vector to form a TF-IDF sparse feature matrix. In text vectorization processing, the final output of all feature extraction methods is expressed as a feature matrix in mathematical form.
[0029] Step 3.3 Determine the optimal number of clusters K: Use the elbow rule and silhouette coefficient to determine the optimal number of clusters K to ensure the best clustering results.
[0030] Step 3.4 T-SNE Dimensionality Reduction Weights and Cluster Density Calculation: The T-SNE method is used to reduce the dimensionality of high-dimensional features, and the optimal number of clusters K selected in Step 3.3 is used to execute the clustering algorithm to generate K different clusters; the density within each cluster is calculated to evaluate the clustering quality and distribution.
[0031] Step 3.5 Resampling to Generate Seed Question-Answer Pairs: Based on the coordinate distribution of each cluster in the reduced-dimensional space from Step 3.4, the cluster density of each cluster is calculated. The core question-answer pairs from Step 2 are then resampled according to the cluster density to ensure data diversity. Finally, new question-answer pairs are generated and saved as question-answer pair text. The cluster density is the ratio of the number of samples within each cluster to the metric value of the space covered by that cluster.
[0032] Step 4. Retrieval Enhancement to Reconstruct Question-Answer Pairs: Using the comprehensive text database formed in Step 1 as a vector database, retrieval enhancement generation techniques are used to reconstruct the seed question-answer pairs generated in Step 3 to ensure the accuracy and diversity of the question-answer pairs. The specific operations for retrieval enhancement to reconstruct question-answer pairs are as follows: Step 4.1: Establish a vector database: Perform text segmentation on the comprehensive text database from Step 1, and use the model provided by Hugging Face to create a vector database for fast retrieval and comparison.
[0033] Step 4.2 Question Rewriting: After vectorizing the seed questions in the seed question-answer pair, retrieve documents related to the seed questions from the vector database. Improve the accuracy of the retrieval by rewriting the questions, and integrate the retrieval results of different questions. Sort the documents according to the semantic relevance between the rewritten seed questions and the retrieved documents and the information importance of the retrieved documents in the comprehensive text database in Step 1. Select the top M positions after sorting to ensure that the relevant corpus of the top M positions in the re-sorting sequence is covered as the technical boundary of the retrieval scope. Step 4.3: Use the language model to generate question-answer pairs for expert annotation: Using the documents related to the seed questions retrieved in Step 4.2 as reference materials, when using the language model to improve the quality of the question-answer pairs to generate the annotated pairs, ensure the accuracy of the question-answer pairs and improve the quality of the original pairs. The text related to the rewritten seed question-answer pairs in Step 4.2 is used as reference materials to generate the annotated question-answer pairs.
[0034] Step 5. Manual Annotation of Question-Answer Pairs: Construct a web annotation system, build a webpage, import the reconstructed question-answer pairs, and allow domain experts to annotate and evaluate the newly generated question-answer pairs through a specific webpage interface. Experts provide annotation feedback and evaluation on the website. Based on the improved quality of the question-answer pairs according to the expert annotation feedback, the reconstructed seed question-answer pairs are provided to humans for evaluation. Human feedback information is collected in a timely manner, and this feedback information is processed using a language model for improving the quality of question-answer pairs to generate a detailed user feedback table, including different user opinions, providing a data foundation for subsequent optimization.
[0035] The manually annotated question-and-answer pairs allow domain experts to annotate and evaluate the question-and-answer pairs obtained in step 4 through a designated web interface. An evaluation mechanism is introduced based on user feedback forms to measure the quality of expert feedback and select the best feedback to improve the question-and-answer pairs. The system comprehensively considers consistency, constructiveness, response efficiency, and satisfaction rate, quantifies the score of each expert, selects the best feedback suggestions to implement the question-and-answer pair improvements, and forms the final question-and-answer pair library. This ensures that the question-and-answer system is continuously iterated and updated, providing high-quality answers to users.
[0036] like Figure 1 and Figure 2As shown, this invention selects a language model for improving the quality of question-answer pairs after comprehensive evaluation. By cleaning and reducing the duplication of different types of data from various channels, a comprehensive text database of the objects involved in the question-answer pairs is finally constructed. After generating original question-answer pairs based on the database by calling the language model, the most representative and diverse seed question-answer pairs are selected by using TF-IDF and K-Means clustering methods. After reconstructing the seed question-answer pairs using retrieval enhancement generation technology, domain experts provide annotation feedback on the reconstructed seed question-answer pairs using retrieval enhancement generation technology, further improving the quality of question-answer pairs.
[0037] Regarding the Q&A for quality improvement assessment, such as Figure 3 , Figure 4 , Figure 5 As shown.
[0038] Figure 3 The difference in matching degree between the original answers (hollow circles) and the new answers (solid squares) is visually illustrated using a scatter plot. The horizontal axis represents the sample number, and the vertical axis represents the matching degree with the standard answer. The plot shows that the matching degree of the new answers is generally higher than that of the original answers. Specifically, the average matching degree of the new answers is 0.79, while the average matching degree of the original answers is only 0.25. It is worth noting that the new answers not only have a higher average matching degree, but their distribution is also more concentrated, indicating that the answers generated by the new method have higher stability and reliability.
[0039] Figure 4 A bar chart was used to compare the distribution of the original answers (lined) and the new answers (solid) across different scoring intervals. The horizontal axis represents the scoring interval, and the vertical axis represents the sample size. The chart shows that the scoring distribution of the new answers is significantly better than that of the original answers. Specifically, the average score of the new answers is 0.59, while the average score of the original answers is 0.23. The scoring distribution of the new answers is concentrated in the higher scoring interval (0.5 to 1.0), while the scoring distribution of the original answers is concentrated in the lower scoring interval (0.0 to 0.5). The concentration of the new answers in the higher scoring interval reflects that the new method can significantly improve the quality of the answers while reducing the generation of low-quality answers. The scoring distribution of the new answers is more concentrated, and in most samples, the new answers score higher than the original answers.
[0040] Figure 5The chart shows a comparison of key technical indicators between the old and new answers. The graph details the differences between the original and new answers in terms of mean, standard deviation, and significance test (p-value). The new answer has a mean of 0.59, while the original answer has a mean of only 0.23, an increase of +0.37. This indicates that the overall score of the new answer is significantly higher than that of the original answer. The standard deviation of the new answer is 0.29, while the standard deviation of the original answer is 0.13, a slight increase (+0.16). This indicates that the score distribution of the new answer is more dispersed, but the overall score level is higher. The p-value for the significance test is 1.52e-91, far less than the commonly accepted threshold of 0.05, indicating that the performance difference between the old and new answers is extremely significant and almost impossible to be caused by random error.
[0041] In summary, Figure 3 , Figure 4 , Figure 5 The data strongly supports the significant effect of this invention in improving the quality of question-and-answer pairs. This invention selects language models (such as Qwen1.5-110B-Chat) through a multi-dimensional evaluation mechanism and evaluates the performance of these models based on standard question-and-answer pairs to ensure that the selected language models possess good answering capabilities. It uses a combination of web crawling and field research to acquire topic-related text materials, and then cleans and organizes the data to construct a comprehensive text database. The invention's dynamic prompting mechanism and the combination of TF-IDF and K-Means methods to select seed question-and-answer pairs ensure that the generated question-and-answer pairs are sufficient in number and credible in content. Expert feedback and annotation further optimize the question-and-answer pairs, ensuring continuous iterative updates to provide more accurate and high-quality answers. This invention significantly improves the response efficiency and answer accuracy of question-and-answer systems when handling complex questions.
[0042] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.
Claims
1. A method for improving the quality of question-answer pairs based on retrieval enhancement generation technology, characterized in that, A language model was selected to improve the quality of question-answer pairs. By cleaning and deduplicating various types of data from multiple channels, a comprehensive text database of the objects involved in the question-answer pairs was constructed. After generating original question-answer pairs using the language model, TF-IDF and K-Means clustering algorithms were used to jointly select the most representative and diverse seed question-answer pairs. These seed pairs were then reconstructed using retrieval enhancement generation techniques, and annotation feedback was provided to improve the quality of the question-answer pairs. The specific steps include: Step 1. Select a language model and acquire and clean the text data: Select multiple different natural language model APIs for the question-answer pair generation task; use web crawling tools and field research to collect text data related to the objects involved in the question-answer pairs, remove irrelevant and repetitive content from the text data, and classify and organize the text data according to the semantic logical order to form a comprehensive text database of question-answer pairs. Step 2. Generate question-answer pairs: Use the natural language model to generate several core question-answer pairs that conform to semantic consistency based on the comprehensive text database obtained in Step 1; Step 3. Select seed question-answer pairs: Calculate the word frequency and importance of the core question-answer pairs obtained in Step 2 using the TF-IDF method, convert the word frequency and importance into feature vectors of the corresponding question-answer pairs, perform cluster analysis, and select the most representative and diverse seed question-answer pairs; Step 4. Retrieval Enhancement Generation and Reconstruction of Question-Answer Pairs: Using the comprehensive text database formed in Step 1 as a vector database, retrieval enhancement generation technology is used to reconstruct the seed question-answer pairs to ensure the accuracy and diversity of the question-answer pairs; Step 5. Manually annotate question-answer pairs: Build a network annotation system to provide the reconstructed seed question-answer pairs to humans for evaluation and collect human feedback in a timely manner.
2. The method for improving the quality of question-answer pairs based on retrieval enhancement generation technology according to claim 1, characterized in that, In step 1, the specific steps for selecting the language model are as follows: Step 1.1: Construction of Standard Question Answering Dataset: Based on the domain and objectives of question answering pair construction, a standard question answering dataset was established as a benchmark test set, and multiple candidate language models were selected to answer questions based on the given standard question answering dataset; Step 1.2 Comprehensive evaluation and selection of language models: Use a comprehensive evaluation method to quantify all language models and select the language model used to improve the quality of question-answer pairs.
3. The method for improving the quality of question-answer pairs based on retrieval enhancement generation technology according to claim 2, characterized in that, The comprehensive evaluation method uses each model to score the answers generated by other models from the following dimensions: accuracy, completeness, relevance, clarity, and politeness, and calculates the mean score, standard deviation, Kolmogorov-Smirnov (KS) statistic, and p-value for each language model.
4. The method for improving the quality of question-answer pairs based on retrieval enhancement generation technology according to claim 1, characterized in that, In step 2, the core question-and-answer pair generation process is as follows: Step 2.1 Text Segmentation: The input text is segmented according to the newline character to form several independent paragraphs. The segmented independent paragraphs are used as input, and a question-answer generation task is performed using a preset language model to obtain the initial question-answer pairs corresponding to the content of each paragraph. Step 2.2 Dynamic Prompts: Calculate the corresponding number of question-and-answer generated N based on the character length L of each independent paragraph, where... C is the preset word count step constant; the paragraph content and the number of generated words N are filled into the preset prompt template to construct targeted dynamic prompts and ensure the high quality and relevance of the generated question-and-answer pairs; Step 2.3 Generate core question-answer pairs: Based on the paragraph length of the input text, the language model used to improve the quality of question-answer pairs generates a number of high-quality core question-answer pairs that match the paragraph length, and saves them as question-answer pair text.
5. The method for improving the quality of question-answer pairs based on retrieval enhancement generation technology according to claim 1, characterized in that, In step 3, the process of selecting seed question-answer pairs is as follows: Step 3.1 Remove stop words and perform word segmentation: Filter the core question-answer pairs obtained in Step 2 for stop words, merge the questions and answers, and then perform word segmentation to improve the relevance of the search. Step 3.2 Feature Extraction and Vectorization: The TF-IDF method is applied to convert the word segmentation results into vectors. The term frequency (TF) of each word in a single question-answer pair and the inverse document frequency (IDF) in the entire question-answer pair set are calculated. The TF and IDF are multiplied to obtain the weight score of each word to measure the importance of the word. Based on the weight score, each question-answer pair is converted into a feature vector to form a TF-IDF sparse feature matrix. Step 3.3 Determine the optimal number of clusters K: Use the elbow rule and silhouette coefficient to determine the optimal number of clusters K to ensure the best clustering results; Step 3.4 T-SNE Dimensionality Reduction Weights and Cluster Density Calculation: The T-SNE method is used to reduce the dimensionality of high-dimensional features, and the optimal number of clusters K selected in Step 3.3 is used to execute the clustering algorithm to generate K different clusters; the density within each cluster is calculated to evaluate the clustering quality and distribution. Step 3.5 Resampling to generate seed question-answer pairs: Based on the coordinate distribution of each cluster in the reduced-dimensional space in Step 3.4, calculate the cluster density of each cluster, and resample the core question-answer pairs in Step 2 according to the cluster density to ensure data diversity. Finally, new question-answer pairs are generated and saved as question-answer pair text.
6. The method for improving the quality of question-answer pairs based on retrieval enhancement generation technology according to claim 1, characterized in that, In step 4, the specific operations for retrieving and enhancing the generation of reconstructed question-answer pairs are as follows: Step 4.1 Establish a vector database: Perform text segmentation on the comprehensive text database from Step 1, and use the model provided by HuggingFace to create a vector database for fast retrieval and comparison; Step 4.2 Question Rewriting: After vectorizing the seed questions in the seed question-answer pair, retrieve documents related to the seed questions from the vector database. Improve the accuracy of the retrieval by rewriting the questions, and integrate the retrieval results of different questions. Sort the documents according to the semantic relevance between the rewritten seed questions and the retrieved documents and the information importance of the retrieved documents in the comprehensive text database in Step 1. Select the top M positions after sorting to ensure that the relevant corpus of the top M positions in the re-sorting sequence is covered as the technical boundary of the retrieval scope. Step 4.3 Call the language model to generate question-answer pairs for expert annotation: Use the documents related to the seed questions retrieved in Step 4.2 as reference materials. When calling the language model used to improve the quality of question-answer pairs to generate question-answer pairs to be annotated, ensure the accuracy of the question-answer pairs and improve the quality of the original question-answer pairs.
7. The method for improving the quality of question-answer pairs based on retrieval enhancement generation technology according to claim 1, characterized in that, In step 5, the manually labeled question-and-answer pairs allow domain experts to annotate and evaluate the question-and-answer pairs obtained in step 4 through a set web interface. An evaluation mechanism is introduced based on the user feedback form to measure the quality of expert feedback and select the best feedback to improve the question-and-answer pairs to be labeled, thus forming the final question-and-answer pair library.