Method for automatically constructing question-answer pairs in vertical field based on large model
Through an automatic construction method of vertical field question and answer pairs based on a large model, the problem of poor quality of data set generation and question and answer pair generation in the existing technology is solved, high-quality, multi-format question and answer pair generation is achieved, and labor costs are reduced. .
Patent Information
- Application Number
- CN202411431514.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-14
- Publication Date
- 2025-01-24
AI Technical Summary
Large models in existing verticals have problems with training dataset generation, Q&A pair generation quality, and supported text formats, resulting in limited performance and poor data quality.
A method for automatically constructing question and answer pairs in vertical fields based on large models is proposed, including obtaining data in multiple formats for conversion, text slicing, generating question and answer pairs based on LLM, clear deduplication filtering, fine-tuning and testing to ensure the quality and quality of question and answer pairs. Format diversity.
It realizes the generation of high-quality vertical field question and answer pairs, supports multiple text formats, reduces labor costs, and improves data quality and generation efficiency.
Smart Images
Figure CN119357331A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of general question answering, and in particular to a method for automatically constructing vertical field question answering pairs based on a large model. Background Art
[0002] The reason why large models perform well in the field of general question answering is mainly due to the extensive and rich data and knowledge they learn during the pre-training stage. This enables them to deeply understand and generate text content covering a variety of different topics. However, although these general large models can demonstrate excellent capabilities in handling daily conversations and general questions, they often encounter some challenges when faced with domain-specific tasks.
[0003] This is mainly due to the professional terms, concepts, and contexts contained in specific fields, which may not be fully covered and learned in the pre-training data of large models. Therefore, when the model needs to handle tasks in these professional fields, its performance may be affected to a certain extent.
[0004] The disadvantages of the existing solutions are:
[0005] First, the existing methods for generating training data sets for large models in vertical fields are time-consuming and labor-intensive, with high labor costs and poor data quality.
[0006] Second, the method of generating question-answer pairs based on unsupervised datasets does not segment the text blocks according to completed topics, resulting in poor quality of subsequent question-answer pair generation.
[0007] Third, the current question-answer generation supports a single format, and most of them only support existing text documents. Summary of the invention
[0008] In view of the above problems, the present invention is proposed to provide a method for automatically constructing vertical domain question-answer pairs based on a large model, which overcomes the above problems or at least partially solves the above problems.
[0009] According to one aspect of the present invention, a method for automatically constructing vertical field question-answer pairs based on a large model is provided, and the construction method comprises:
[0010] Obtain data in various formats, convert them into plain text data, and perform text segmentation on the plain text data;
[0011] Generate question-answer pairs based on LLM;
[0012] The generated question-answer pairs are filtered for clear deduplication, and the LLM of 7B is fine-tuned and tested using the filtered question-answer pairs for clear deduplication. The question-answer pairs are checked to obtain question-answer pairs.
[0013] Optionally, the acquiring of data in multiple formats specifically includes:
[0014] Original text, knowledge graph and Excel formats;
[0015] When texts in various formats are input, the texts are formatted and converted into a unified format of the original text.
[0016] Optionally, the text segmentation of the plain text data specifically includes:
[0017] Divide the text into chunks based on the delimiter for the first time;
[0018] Set the size of the text block to chunk_size1, and concatenate the results of the first block division until chunk_size1 is reached, and then get the result of the second block division;
[0019] For the result of the second block division, the sentence vector representation of each text block is obtained based on LLM, and the cosine similarity of adjacent text blocks is calculated based on the sentence vector;
[0020] A similarity threshold is set, and the size of the text block chunk_size2 is set. Based on the similarity threshold and the size of the text block chunk_size2, it is determined whether to merge adjacent text blocks, and the result of the third block division is obtained to obtain the final text block result.
[0021] Optionally, the generating of question-answer pairs based on LLM specifically includes:
[0022] Use prompt according to the text type, input the prompt and the original text block into LLM, generate question 1, input the original text block and question 1 combined with the corresponding prompt into LLM, generate answer 1;
[0023] Input answer 1 and the corresponding prompt into LLM to generate question 2;
[0024] Obtain a set of question-answer pairs 1 and a set of question-answer pairs 2;
[0025] Take the intersection of question 1 set in question-answer pair 1 set and question 2 set in question-answer pair 2 set based on similarity. If the similarity threshold is reached between the two, they are retained, otherwise they are filtered out;
[0026] LLM uses GPT-4 or open source cutting-edge large models to generate high-quality question-answer pairs.
[0027] Optionally, performing clear deduplication filtering on the generated question-answer pairs specifically includes:
[0028] Use the SimHash algorithm to remove duplicates from question and answer pairs;
[0029] Quality filtering, calculate the perplexity of the question and answer pair, and filter out semantically unsmooth and disordered text by calculating the value of text perplexity.
[0030] Optionally, the quality filtering method includes:
[0031] Train an n-gram language model, use the n-gram language model to calculate the perplexity of the text, and set the perplexity threshold to filter out low-quality texts with perplexity greater than the threshold;
[0032] Through perplexity evaluation, we filter out texts that the language model cannot accurately understand, and delete sentences containing punctuation, HTML, links, and sensitive words.
[0033] Sensitive content filtering: the collected data is filtered by adopting specific filtering rules or text anti-spam classification models.
[0034] Optionally, the fine-tuning of the LLM of 7B specifically includes:
[0035] LLM fine-tuning is the process of further training a pre-trained large model using a dataset in a specific field, aiming to optimize the performance of the model on a specific task so that the model can better adapt to and complete tasks in a specific field;
[0036] Through the generated question-answer pairs, the LLM of about 7B is fine-tuned using the full-quantity fine-tuning, QLoRA, P-Truning, BitFit, Prompt Tuning, and AdapterTuning methods.
[0037] Optionally, the LLM of 7B is tested, and the question-answer pair is checked to obtain the question-answer pair, which specifically includes:
[0038] Use the LLM fine-tuned based on question-answer pairs to evaluate the test set in the vertical field, and compare the evaluation results with those of the original LLM without fine-tuning to see whether the evaluation results of the fine-tuned LLM are improved, so as to judge whether the quality of the generated question-answer pairs is acceptable;
[0039] If there is an improvement, it means that the generated question-answer pairs are of high quality and are retained; otherwise, adjust the prompt, separator, chunk_size1, chunk_size2, and regenerate the question-answer pairs.
[0040] The present invention provides a method for automatically constructing vertical field question-answer pairs based on a large model, the construction method comprising: obtaining data in various formats, converting them into plain text data, and performing text segmentation on the plain text data; generating question-answer pairs based on LLM; performing clear deduplication filtering on the generated question-answer pairs, fine-tuning and testing the LLM of 7B with the question-answer pairs after clear deduplication filtering, checking the question-answer pairs, and obtaining question-answer pairs. It supports multiple text formats, from text block segmentation to question-answer pair generation, and then to question-answer pair filtering and screening, thus realizing end-to-end text question-answer pair generation, and further ensuring the quality of the generated question-answer pairs.
[0041] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0043] Figure 1 A flowchart of a method for automatically constructing vertical field question-answer pairs based on a large model provided by an embodiment of the present invention;
[0044] Figure 2 A flowchart of a method for automatically constructing vertical field question-answer pairs based on a large model provided by an embodiment of the present invention;
[0045] Figure 3 A flowchart of text slicing provided by an embodiment of the present invention;
[0046] Figure 4 A flowchart of generating question-answer pairs based on LLM provided in an embodiment of the present invention;
[0047] Figure 5 A flowchart of quality monitoring and filtering of question and answer pairs provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0048] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0049] The terms "comprises" and "having" and any variations thereof in the description embodiments, claims and drawings of the present invention are intended to cover non-exclusive inclusions, for example, including a series of steps or units.
[0050] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.
[0051] The automatic construction method of overall vertical field question-answer pairs is as follows Figure 1 As shown, data in various formats are obtained, converted into plain text data, the plain text data is segmented, and question-answer pairs are generated based on LLM. The generated question-answer pairs are then filtered for clear deduplication. The LLM of about 7B is fine-tuned and tested using the question-answer pairs after clear deduplication. This is to further check the question-answer pairs and obtain the final high-quality question-answer pairs.
[0052] The text block segmentation part first segments the text based on the separator and the size of the text block, then calculates the similarity between adjacent text blocks, and then decides whether to merge adjacent text blocks based on the set similarity threshold to obtain the final text block.
[0053] The part of generating question-answer pairs based on LLM is to first generate question-answer pair 1 based on LLM, then generate question-answer pair 2, and then take the intersection of the questions of question-answer pair 1 and question-answer pair 2 to obtain the final question-answer pair generated by LLM. The purpose of taking the intersection here is to filter the first-level question-answer pairs to ensure the quality of the generated question-answer pairs.
[0054] In the part of the question-answer pair quality check and filtering, the questions are first filtered through deduplication, clarity, and text anti-spam. The filtered question-answer pairs are then used to fine-tune the LLM of the 7B function. The effects before and after fine-tuning are then tested using a vertical field test set to determine the quality of the generated question-answer pairs.
[0055] The overall simplified method flow is as follows Figure 2 shown.
[0056] Text input and conversion in various formats, including: Supporting text input in multiple formats, including original text, knowledge graph, and Excel format. After text in various formats is input, format conversion is performed to convert it into a unified original text format. This is used in subsequent modules.
[0057] like Figure 3 As shown, the text is segmented into:
[0058] In order to ensure the integrity of the sentence, first set the separator, such as line break, period, exclamation mark. Then divide the text into blocks for the first time based on the separator. Then set the size of the text block chunk_size1, and then splice the results of the first block until chunk_size1 is reached. In this way, the result of the second block is obtained.
[0059] For the result of the second block division, the sentence vector representation of each text block is obtained based on LLM, and the cosine similarity of adjacent text blocks is calculated based on the sentence vector.
[0060] Set a reasonable similarity threshold, set the size of the text block chunk_size2, and decide whether to merge adjacent text blocks based on the threshold and chunk_size2 to obtain the result of the third block division, that is, the final text block result.
[0061] The process of generating question-answer pairs based on LLM is as follows Figure 4 shown.
[0062] Different prompts are used for different text types. Select the appropriate prompt based on the text type, input the prompt and the original text block into the LLM to generate question 1, and then input the original text block and question 1 combined with the corresponding prompt into the LLM to generate answer 1.
[0063] Then, answer 1 and the corresponding prompt are input into LLM to generate question 2. In this way, we get the set of question-answer pairs 1 and the set of question-answer pairs 2.
[0064] Then, based on the similarity, we take the intersection of question 1 set in question-answer pair 1 and question 2 set in question-answer pair 2. That is, those that meet the similarity threshold are considered high-quality and are retained, otherwise they are filtered out. This is done to ensure the generation of high-quality question-answer pairs, and is also the first filtering operation for question-answer pairs.
[0065] The LLM here can use GPT-4 or an open source cutting-edge large model to ensure that high-quality question-answer pairs can be generated to guarantee a series of subsequent operations.
[0066] The question and answer quality monitoring and filtering part is mainly used to achieve further clear filtering of the question and answer pairs, and to ensure the quality of the question and answer pairs through layers of filtering and screening.
[0067] like Figure 5 As shown, the question and answer pair cleaning filter includes:
[0068] (1) Data deduplication, including:
[0069] The question and answer pairs are deduplicated using the SimHash algorithm.
[0070] The main idea of the SimHash algorithm is dimensionality reduction, mapping high-dimensional feature vectors into low-dimensional feature vectors, and determining whether the articles are repeated or highly similar through the Hamming Distance of the two vectors.
[0071] After the questions in the question-answer pair are processed by the SimHash algorithm, the similarity between them can be calculated by comparing the Hamming distance of the SimHash values of the two questions. Here we set the similarity threshold of the two questions to 4, and calculate the Hamming distance between the two questions by performing bit operations on the SimHash values of the two questions. For SimHash values less than or equal to 4, they are considered to be the same text, while those greater than 4 are considered to be different questions. In this way, when comparing the number of different bits in the Simhash values of the two, the smaller the Hamming distance, the more similar the texts are.
[0072] (2) Quality filtering, including:
[0073] Calculate the perplexity of the question-answer pair and filter based on the text perplexity. By calculating the value of the text perplexity, semantically unsmooth texts and some disordered texts are filtered out.
[0074] Specific method: Train an n-gram language model, use the model to calculate the perplexity of the text, and set a perplexity threshold to filter out low-quality texts with perplexity greater than the threshold. Through perplexity evaluation, you can filter out texts that the language model cannot accurately understand, which are often of poor quality. Delete sentences with punctuation, HTML, links, and sensitive words.
[0075] Perplexity and special symbol screening are important means to improve data quality.
[0076] (3) Sensitive content filtering, including:
[0077] In addition to removing low-quality content, the collected data may also include toxic content or privacy information, which requires further more detailed filtering and processing.
[0078] Filtering is performed by adopting specific filtering rules or text anti-spam classification models.
[0079] LLM fine-tuning includes: LLM fine-tuning is the process of further training a pre-trained large model using a dataset in a specific field, aiming to optimize the performance of the model on a specific task so that the model can better adapt to and complete tasks in a specific field.
[0080] Fine-tune the LLM of about 7B through the previously generated question-answer pairs. You can use full-volume fine-tuning, QLoRA, P-Truning, BitFit, Prompt Tuning, Adapter Tuning, etc. The specific method depends on the business scenario, server resources, and the number of question-answer pairs.
[0081] The fine-tuned LLM is tested to evaluate question-answer pairs including:
[0082] Use the LLM fine-tuned based on the question-answer pair to evaluate the test set in the vertical field, and compare the evaluation results of the original LLM without fine-tuning to see whether the evaluation results of the fine-tuned LLM have improved, so as to judge whether the quality of the generated question-answer pair is up to standard. If there is an improvement, it means that the generated question-answer pair is of high quality and is retained; otherwise, adjust the prompt, separator, chunk_size1, chunk_size2, etc., and regenerate the question-answer pair.
[0083] Beneficial effects: The present invention proposes a method and device for automatically constructing vertical field question-answer pairs based on a large model, and proposes an effective text block segmentation method, which ensures the complete topic segmentation of the text block.
[0084] The present invention combines character segmentation, text block size setting, adjacent text similarity calculation and splicing multi-step processing to effectively ensure the integrity of the text block theme.
[0085] The present invention proposes a method for generating high-quality question-answer pairs based on a large model (LLM). This method generates question-answer pairs twice and takes the intersection method to perform the first filtering for the generation of high-quality question-answer pairs. Secondly, after deduplication filtering of question-answer pairs and fine-tuning test of LLM with 7B effect, the quality of question-answer pairs is further guaranteed. It is a fully automatic method for constructing vertical field data sets without labor costs, and it has undergone layers of screening to ensure the quality of the generated question-answer pairs.
[0086] Supports the generation of question-answer pairs based on texts in various formats, such as documents, knowledge graphs, Excel tables, etc.
[0087] In summary, the present invention provides a method for automatically constructing vertical field question-answer pairs based on a large model, which supports multiple text formats, from text block segmentation to question-answer pair generation, and then to question-answer pair filtering and screening, thereby realizing end-to-end text question-answer pair generation and further ensuring the quality of the generated question-answer pairs.
[0088] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for automatically constructing vertical field question-answer pairs based on a large model, characterized in that: The construction method comprises: Obtain data in various formats, convert them into plain text data, and perform text segmentation on the plain text data; Generate question-answer pairs based on LLM; The generated question-answer pairs are filtered for clear deduplication, and the LLM of 7B is fine-tuned and tested using the filtered question-answer pairs for clear deduplication. The question-answer pairs are checked to obtain question-answer pairs.
2. According to the method of automatic construction of vertical field question-answer pairs based on a large model according to claim 1, it is characterized in that: The acquisition of data in multiple formats specifically includes: Original text, knowledge graph and Excel formats; When texts in various formats are input, the texts are formatted and converted into a unified format of the original text.
3. According to the method for automatically constructing vertical field question-answer pairs based on a large model in claim 1, it is characterized in that: The text segmentation of the plain text data specifically includes: Divide the text into chunks based on the delimiter for the first time; Set the size of the text block to chunk_size1, and concatenate the results of the first block division until chunk_size1 is reached, and then get the result of the second block division; For the result of the second block division, the sentence vector representation of each text block is obtained based on LLM, and the cosine similarity of adjacent text blocks is calculated based on the sentence vector; A similarity threshold is set, and the size of the text block chunk_size2 is set. Based on the similarity threshold and the size of the text block chunk_size2, it is determined whether to merge adjacent text blocks, and the result of the third block division is obtained to obtain the final text block result.
4. According to the method for automatically constructing vertical field question-answer pairs based on a large model according to claim 1, it is characterized in that: The generation of question-answer pairs based on LLM specifically includes: Use prompt according to the text type, input the prompt and the original text block into LLM, generate question 1, input the original text block and question 1 combined with the corresponding prompt into LLM, generate answer 1; Input answer 1 and the corresponding prompt into LLM to generate question 2; Obtain a set of question-answer pairs 1 and a set of question-answer pairs 2; Take the intersection of question 1 set in question-answer pair 1 set and question 2 set in question-answer pair 2 set based on similarity. If the similarity threshold is reached between the two, they are retained, otherwise they are filtered out; LLM uses GPT-4 or open source cutting-edge large models to generate high-quality question-answer pairs.
5. According to the method for automatically constructing vertical domain question-answer pairs based on a large model according to claim 1, it is characterized in that: The clear deduplication filtering of the generated question-answer pairs specifically includes: Use the SimHash algorithm to remove duplicates from question and answer pairs; Quality filtering, calculate the perplexity of the question and answer pair, and filter out semantically unsmooth and disordered text by calculating the value of text perplexity.
6. According to the method of automatic construction of vertical field question-answer pairs based on a large model in claim 1, it is characterized in that: The quality filtering method comprises: Train an n-gram language model, use the n-gram language model to calculate the perplexity of the text, and set the perplexity threshold to filter out low-quality texts with perplexity greater than the threshold; Through perplexity evaluation, we filter out texts that the language model cannot accurately understand, and delete sentences containing punctuation, HTML, links, and sensitive words. Sensitive content filtering: the collected data is filtered by adopting specific filtering rules or text anti-spam classification models.
7. According to the method of automatic construction of vertical field question-answer pairs based on a large model in claim 1, it is characterized in that: The fine-tuning of the LLM of 7B specifically includes: LLM fine-tuning is the process of further training a pre-trained large model using a dataset in a specific field, aiming to optimize the performance of the model on a specific task so that the model can better adapt to and complete tasks in a specific field; Through the generated question-answer pairs, the LLM of about 7B is fine-tuned using the full-quantity fine-tuning, QLoRA, P-Truning, BitFit, Prompt Tuning, and AdapterTuning methods.
8. According to the method of automatic construction of vertical field question-answer pairs based on a large model in claim 1, it is characterized in that: The LLM of 7B is tested, and the question-answer pair is checked, and the question-answer pair specifically includes: Use the LLM fine-tuned based on question-answer pairs to evaluate the test set in the vertical field, and compare the evaluation results with those of the original LLM without fine-tuning to see whether the evaluation results of the fine-tuned LLM are improved, so as to judge whether the quality of the generated question-answer pairs is acceptable; If there is an improvement, it means that the generated question-answer pairs are of high quality and are retained; otherwise, adjust the prompt, separator, chunk_size1, chunk_size2, and regenerate the question-answer pairs.
Citation Information
Cited By
Data generation method and device, electronic equipment and storage medium
CN120851168A