Question and answer pair generation method and device, electronic equipment and readable storage medium

Abstract

The application provides a question and answer pair generation method and device, electronic equipment and readable storage medium, wherein the method comprises: obtaining a text semantic block to be processed; generating a candidate question and answer pair set corresponding to the text semantic block based on multi-view prompt language; identifying remaining semantic information not covered and generating a supplementary question and answer pair according to the text semantic block and the candidate question and answer pair set, to obtain an expanded question and answer pair set; and performing verification and deduplication processing on the expanded question and answer pair set, and outputting a target question and answer pair dataset. The method guides the completion of the remaining semantic information through the multi-view prompt language, fully mines the multi-level knowledge in a single text semantic block, significantly improves the information coverage and diversity of the question and answer pair, and effectively eliminates illusory content and redundant data through answer verifiability and deduplication filtering processing, thereby guaranteeing the high quality and reliability of the output question and answer pair dataset.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing, and in particular to a question-and-answer pair generation method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] With the rapid development of large-scale language models and intelligent question-answering systems, question-answer pairs have become a core resource for model training, capability optimization, and knowledge enhancement. Currently, the main methods for constructing question-answer pairs include manual annotation, template rule generation, and automatic generation based on language models. Among these, question-answer generation methods based on text semantic segmentation are widely used in scenarios such as retrieval enhancement generation, domain knowledge base construction, and pre-training data augmentation due to their good scalability.

[0003] However, existing technologies still have significant shortcomings in practical applications: First, manual annotation methods are costly and inefficient, making it difficult to support large-scale construction of massive datasets; second, when generating questions and answers directly from entire text segments, the broad semantic scope often results in questions with weak focus and coarse granularity; more critically, existing methods based on text semantic segmentation typically generate only a small number of questions and answers from a single perspective, lacking in-depth mining of the multi-layered and multi-dimensional semantic information implicit within the text semantic segments, leading to the omission of a large amount of effective information. Furthermore, existing evaluation mechanisms are mostly limited to surface-level checks of language fluency or answer correctness, lacking a systematic evaluation of information coverage, and unable to measure whether all key information is covered, whether the coverage of different types of information is balanced, and whether there is duplicate coverage.

[0004] In summary, existing methods often result in a surge in data volume but limited information increment, making it difficult to fully unleash the knowledge value of the original text. Therefore, there is an urgent need for a technical solution that can fully mine multi-level semantic information from semantic blocks of a single text and generate high-coverage question-answer pairs by combining coverage evaluation mechanisms. Summary of the Invention

[0005] This invention provides a question-and-answer pair generation method, apparatus, electronic device, and readable storage medium to solve the problems in existing question-and-answer generation technologies, such as a single perspective, incomplete information coverage, and lack of systematic coverage evaluation, resulting in large data scale but low information increment. It enables the full mining of multi-level semantic information from a single text block to generate question-and-answer pairs with high coverage, high diversity, and semantic completeness.

[0006] On one hand, the present invention provides a question-answer pair generation method, comprising: acquiring a text semantic block to be processed; generating a candidate question-answer pair set corresponding to the text semantic block based on multi-perspective prompts; the multi-perspective prompts include at least two of information type, role setting, and information granularity; identifying uncovered remaining semantic information and generating supplementary question-answer pairs according to the text semantic block and the candidate question-answer pair set, thereby obtaining an extended question-answer pair set; performing verification and deduplication processing on the extended question-answer pair set, and outputting a target question-answer pair dataset.

[0007] Furthermore, the information types include at least two of the following: fact definition, relational reasoning, and process summary, used to cover entity attributes and concept interpretations in the text semantic block; and / or, the role settings include professional roles and ordinary user roles, used to cover causal logic, conditional dependencies, and comparative differences in the text semantic block; and / or, the information granularity includes global semantic granularity and local semantic granularity, used to cover process steps and overall theme in the text semantic block.

[0008] Further, the step of identifying uncovered remaining semantic information and generating supplementary question-answer pairs based on the text semantic block and the candidate question-answer pair set includes: simultaneously inputting the text content of the text semantic block and the candidate question-answer pair set into a large language model; guiding the large language model to compare the text content of the text semantic block with the answer content in the candidate question-answer pair set through a prompt to identify remaining semantic information in the text semantic block that is not covered by the answer content; constructing a targeted generation prompt based on the remaining semantic information; and calling the large language model to generate supplementary question-answer pairs for the remaining semantic information; wherein the answers of the supplementary question-answer pairs are constrained to be restates or summaries of the remaining semantic information.

[0009] Further, the step of identifying uncovered remaining semantic information and generating supplementary question-answer pairs based on the text semantic block and the candidate question-answer pair set is performed iteratively until a preset stopping condition is met, resulting in an expanded question-answer pair set; wherein, the preset stopping condition includes any of the following: no new valid supplementary question-answer pairs are generated for several consecutive rounds; the total number of supplementary question-answer pairs reaches a preset upper limit; the number of iteration rounds reaches a preset upper limit.

[0010] Furthermore, the verification and deduplication process for the extended question-answer pair set includes: performing semantic consistency verification on the supplementary question-answer pairs in the extended question-answer pair set to determine whether the question and answer match; performing answer verifiability verification on the supplementary question-answer pairs in the extended question-answer pair set to determine whether the answer can be verified from the text semantic block; and performing duplicate and similar question-answer filtering on the supplementary question-answer pairs in the extended question-answer pair set to remove semantically duplicated supplementary question-answer pairs.

[0011] Furthermore, when performing answer verifiability verification on the supplementary question-answer pairs in the extended question-answer pair set, if it is determined that the answer contains information that cannot be directly supported or deduced from the text semantic block, the supplementary question-answer pair is determined to be an invalid question-answer pair and is removed.

[0012] Furthermore, the step of obtaining the text semantic block to be processed includes: performing semantic analysis and segmentation on the original text, and dividing the original text into at least one text semantic block according to semantic integrity and length constraints.

[0013] Secondly, the present invention also provides a question-and-answer pair generation apparatus, comprising: a text semantic block acquisition module for acquiring a text semantic block to be processed; a candidate question-and-answer pair generation module for generating a set of candidate question-and-answer pairs corresponding to the text semantic block based on multi-perspective prompts; the multi-perspective prompts include at least two of information type, role setting, and information granularity; a supplementary question-and-answer pair generation module for identifying uncovered remaining semantic information and generating supplementary question-and-answer pairs based on the text semantic block and the set of candidate question-and-answer pairs, iteratively executing until a preset stopping condition is met to obtain an expanded question-and-answer pair set; and a target question-and-answer pair output module for performing verification and deduplication processing on the expanded question-and-answer pair set to output a target question-and-answer pair dataset.

[0014] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the question-answer pair generation method as described above.

[0015] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the question-answer pair generation method as described above.

[0016] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the question-answer pair generation method as described above.

[0017] The question-answer pair generation method provided by this invention obtains the semantic block of the text to be processed and generates a candidate question-answer pair set corresponding to the text semantic block based on multi-perspective prompts. The multi-perspective prompts include at least two of the following: information type, role setting, and information granularity. Then, based on the text semantic block and the candidate question-answer pair set, the remaining uncovered semantic information is identified and supplementary question-answer pairs are generated to obtain an expanded question-answer pair set. Finally, the expanded question-answer pair set is validated and deduplicated to output the target question-answer pair dataset. This method, through multi-perspective prompt guidance and the completion of remaining semantic information, achieves full mining of multi-level knowledge within a single text semantic block, significantly improving the information coverage and diversity of question-answer pairs. At the same time, combined with answer verifiability and deduplication filtering, it effectively eliminates illusory content and redundant data, ensuring the high quality and reliability of the output question-answer pair dataset, which can be directly used for large language model training and knowledge base construction. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the question-and-answer pair generation method provided in an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram of the overall process of the question-answer pair generation method provided in the embodiments of the present invention.

[0021] Figure 3 This is a schematic diagram of the question-and-answer pair generation device provided in an embodiment of the present invention.

[0022] Figure 4 This is a schematic diagram of the physical structure of the electronic device provided in the embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] Figure 1 A flowchart illustrating the question-answer pair generation method provided in an embodiment of the present invention is shown. Figure 1As shown, the method includes: S110, obtaining a text semantic block to be processed; S120, generating a candidate question-answer pair set corresponding to the text semantic block based on multi-perspective prompts; the multi-perspective prompts include at least two of the following: information type, role setting, and information granularity; S130, identifying uncovered remaining semantic information and generating supplementary question-answer pairs based on the text semantic block and the candidate question-answer pair set, to obtain an extended question-answer pair set; S140, performing verification and deduplication processing on the extended question-answer pair set, and outputting the target question-answer pair dataset.

[0025] The following will provide a detailed description of steps S110-S140 and related steps.

[0026] S110, Obtain the text semantic block to be processed.

[0027] The process is straightforward: first, the raw text to be processed is obtained. This raw text includes, but is not limited to, technical documents, product manuals, laws and regulations, academic papers, or web articles. To facilitate subsequent processing, the raw text undergoes preprocessing, such as removing redundant HTML tags, special control characters, and garbled text, unifying full-width and half-width characters, and initially dividing the raw text into natural paragraphs.

[0028] Subsequently, text semantic segmentation is performed. Based on semantic integrity, contextual consistency, and length constraints, the preprocessed original text is divided into at least one text semantic block. Specifically, heading levels, chapter structures, or subheadings in the original text can be used as initial segmentation anchors. For paragraphs exceeding a preset maximum length threshold, they are further split into several sentence units based on sentence boundaries such as periods, question marks, and semicolons. Adjacent sentences are merged using a sliding window method to ensure that each generated text semantic block is semantically relatively self-sufficient, meaning it can independently express a complete knowledge point or narrative unit without relying on preceding or following context. Finally, the output is a sequence of text semantic blocks to be processed, which can be used in subsequent steps.

[0029] The preset maximum length threshold can be set according to the actual situation, such as 512 tokens or 1024 characters, and no specific limit is made here.

[0030] S120, Based on multi-perspective prompts, generate a set of candidate question-answer pairs corresponding to the text semantic block; the multi-perspective prompts include at least two of the following: information type, role setting, and information granularity.

[0031] For each text semantic block obtained in step S110, a multi-perspective generated prompt is constructed to call a large language model to generate a set of candidate question-answer pairs.

[0032] In this embodiment, the multi-perspective prompt integrates at least two of the following: information type, role setting, and information granularity. As a preferred implementation, the multi-perspective prompt integrates all three: information type, role setting, and information granularity.

[0033] The information type perspective guides the model to focus on different types of knowledge within semantic blocks of text. For example, when focusing on factual definitions, prompts guide the model to extract entity attributes and concept interpretations, such as "What is photosynthesis?"; when focusing on relational reasoning, prompts guide the model to uncover causal logic and conditional dependencies, such as "Under what conditions does water freeze?"; and when focusing on process summaries, prompts guide the model to summarize process steps or overall theme, such as "What does this article mainly introduce?".

[0034] Meanwhile, a role-setting perspective is incorporated to enhance diversity. For example, role instructions such as "assuming you are a domain professional" or "assuming you are a regular user" are inserted into the prompts, allowing the generated questions to differentiate in terms of wording and focus. In addition, a granular information perspective is used to construct summary questions targeting the overall semantic block of the text and factual questions targeting specific details within the block.

[0035] This embodiment introduces multi-perspective prompts, enabling the large language model to output an initial set of candidate question-answer pairs. This set contains preliminary question-answer results obtained from mining text semantic blocks from multiple dimensions and levels.

[0036] S130, based on the text semantic block and the candidate question-answer pair set, identify the remaining semantic information that has not been covered and generate supplementary question-answer pairs to obtain an extended question-answer pair set.

[0037] This step improves the information coverage of the text semantic block through an iterative completion mechanism. Specifically, the text content corresponding to the text semantic block and the candidate question-answer pair set generated in step S120 are input into the large language model. Through carefully designed prompts, the large language model is guided to perform semantic difference comparison, that is, to compare and analyze the original semantic content of the text semantic block with the semantic content covered by each answer in the candidate question-answer pair set, and identify the remaining semantic information in the semantic text block that has not yet been effectively covered by any candidate question-answer pair answer, such as a missing technical parameter, an unmentioned causal relationship, or a specific operational step.

[0038] After identifying the remaining semantic information, a targeted prompt is constructed based on this information. The large language model is then invoked again to generate supplementary question-and-answer pairs for this remaining semantic information. The answers to these supplementary pairs are strictly constrained to be restates or summaries of the remaining semantic information to prevent the model from generating illusions. The newly generated supplementary question-and-answer pairs are then added to the candidate question-and-answer pair set, completing one iteration.

[0039] The above iterative process continues until a preset stopping condition is met. This preset stopping condition includes any of the following: no new valid supplementary question-answer pairs are generated for several consecutive rounds, indicating that information is nearing saturation; the total number of supplementary question-answer pairs reaches a preset upper limit; or the number of iteration rounds reaches a preset upper limit. After the iteration terminates, the expanded candidate question-answer pair set is output, i.e., the expanded question-answer pair set.

[0040] The preset quantity limit and preset round limit can be set according to actual needs, and no specific limit is set here.

[0041] S140, Perform validation and deduplication on the extended question-answer pair set, and output the target question-answer pair dataset.

[0042] After obtaining the extended question-and-answer pair set in step S130, it needs to be validated and deduplicated in multiple dimensions to ensure the quality of the output question-and-answer pairs.

[0043] Specifically, firstly, a semantic consistency check is performed to determine whether the answer in each question-and-answer pair in the expanded question-and-answer pair set truly answers the question, eliminating invalid questions and answers that are irrelevant or logically broken. Next, an answer verifiability check is performed, comparing the answer content of each question-and-answer pair in the expanded question-and-answer pair set with the original semantic content of the text semantic block. If it is determined that the answer contains information that cannot be directly supported or derived from the text semantic block, such as fabricated data or subjectively conjectured conclusions, then the question-and-answer pair is determined to be invalid and eliminated.

[0044] Subsequently, duplicate similar question-and-answer filtering is performed. By calculating the semantic similarity between each question-and-answer pair in the expanded question-and-answer pair set, for example, by calculating cosine similarity using text embedding vectors, if duplicate question-and-answer pairs are found with semantic similarity higher than a preset threshold, only the one with richer information or more standardized expression is retained, and the rest of the redundant items are removed.

[0045] After the above verification and deduplication processes, the final output is the target question-answer pair dataset. The question-answer pairs in this dataset have the characteristics of high coverage, high diversity, semantic consistency and traceable source. They can be directly used for supervised fine-tuning training of large language models, knowledge base construction of retrieval enhancement generation systems or corpus enhancement of intelligent question-answering systems.

[0046] In this embodiment, a candidate question-and-answer pair set is generated based on multi-perspective prompts, whereby the text semantic block to be processed is acquired. The multi-perspective prompts include at least two of the following: information type, role setting, and information granularity. Then, based on the text semantic block and the candidate question-and-answer pair set, the remaining uncovered semantic information is identified and supplementary question-and-answer pairs are generated, resulting in an expanded question-and-answer pair set. Finally, the expanded question-and-answer pair set is validated and deduplicated to output the target question-and-answer pair dataset. This method, through multi-perspective prompt guidance and iterative completion of remaining semantic information, fully mines multi-level knowledge within a single text semantic block, significantly improving the information coverage and diversity of question-and-answer pairs. Simultaneously, combined with answer verifiability and deduplication filtering, it effectively eliminates illusory content and redundant data, ensuring the high quality and reliability of the output question-and-answer pair dataset, which can be directly used for large language model training and knowledge base construction.

[0047] Based on the above embodiments, the following will further describe in detail the types and functions of the multi-perspective prompts in step S120.

[0048] Information types include fact definition, relational reasoning, and process summary, used to cover at least two of the entity attributes and concept definitions in the text semantic block; and / or, role settings include professional roles and ordinary user roles, used to cover causal logic, conditional dependencies, and comparative differences in the text semantic block; and / or, information granularity includes global semantic granularity and local semantic granularity, used to cover process steps and overall theme in the text semantic block.

[0049] It is easy to understand that this embodiment uses structured multi-perspective prompt word engineering to mine text semantic blocks from a heterogeneous dimension in order to improve the completeness of the coverage of text semantic information by question-answer pair data.

[0050] In one specific embodiment, the semantic mining and question answering generation process based on information type perspective is described in detail.

[0051] This embodiment addresses the heterogeneous knowledge structure within text semantic blocks by configuring different prompt word templates, forcing the large language model to focus on three types of information: fact definition, relational reasoning, and process summary, thereby achieving full-level coverage of text semantic blocks.

[0052] When generating fact-definition question-answer pairs, prompts are constructed for static knowledge units within text semantic blocks, with extraction and definition as the core. For example, when the model recognizes proper nouns or entities such as photosynthesis and Newton within a text semantic block, the prompts guide the model to generate question-answer pairs such as "What is photosynthesis?" or "In which year did Newton publish *Mathematical Principles of Natural Philosophy*?". The fact-definition perspective is primarily used to cover entity attributes, concept definitions, and basic data within text semantic blocks, ensuring that fundamental semantic content is not lost.

[0053] When generating relational reasoning question-and-answer pairs, prompts are constructed based on the dynamic logical connections within text semantic blocks, focusing on analysis and inference. For example, when a text semantic block contains statements such as "glaciers are melting due to rising temperatures" or "water freezes at 0 degrees Celsius," the prompts guide the model to capture the causal logic and conditional dependencies within, generating question-and-answer pairs such as "What are the reasons for the rise in global average temperature?" or "Under what conditions does water freeze?" The relational reasoning perspective is used to uncover the logical relationships implicit between the lines, rather than simply restating the original words and phrases.

[0054] When generating process summary question-and-answer pairs, prompts are constructed based on the overall architecture or process evolution of the text semantic block, focusing on summarizing and listing key points. For example, when the text semantic block describes a series of operational steps or a lengthy discussion, the prompts guide the model to generate question-and-answer pairs such as "What steps are typically included in data preprocessing?" or "What is the main content of this article?". The process summary perspective is used to capture global semantics, preventing the model from getting bogged down in local details and neglecting the overall theme.

[0055] In another specific embodiment, the semantic mining and question-answering generation process based on role setting perspective is described in detail.

[0056] This embodiment injects role setting parameters to change the stance and focus preferences of the large language model when generating questions, thereby deriving diverse forms of expression on the same information type.

[0057] When generating question-and-answer pairs from a professional perspective, the prompt should include a prompt such as "Assuming you are a domain expert / engineer / auditor," guiding the model to ask questions from the perspectives of professionalism, compliance, underlying principles, and boundary conditions. For example, given the same text about drug use, a professional might focus on questions like "What is the half-life of this drug?" or "What are the contraindications?"

[0058] When generating question-and-answer pairs from the perspective of an ordinary user, the prompt should include "Assuming you are an ordinary user / consumer encountering this content for the first time," guiding the model to ask questions from the perspectives of usability, intuitiveness, common misconceptions, and application effectiveness. For example, given the same text about medication, an ordinary user might focus on "What are the side effects of taking this medication?" or "When should I take it?"

[0059] This embodiment enables the model to examine the same semantic block of text from the perspectives of different stakeholders through this role switching, thereby significantly improving the diversity and breadth of the question-and-answer pair data and avoiding all questions presenting a single role's didactic tone.

[0060] In another specific embodiment, the semantic mining and question answering generation process based on the information granularity perspective is described in detail.

[0061] This embodiment balances global and local semantic granularity by controlling the scope of attention specified in the prompts, ensuring that the generated question-and-answer pairs include both macro-level context and micro-level details.

[0062] At the global semantic granularity, the prompts indicate that the model ignores specific details and focuses on the core viewpoints and main conclusions of the paragraph, generating question-and-answer pairs that cover the main theme of the entire text semantic block.

[0063] At the local semantic granularity, the prompting model focuses on specific data, specific noun explanations, and individual event descriptions in the text, generating question-and-answer pairs for specific segments within the text semantic block.

[0064] In this embodiment, when generating the candidate question-answer pair set corresponding to the text semantic block based on multi-perspective prompts, the aforementioned single perspective is not used in isolation, but rather a combination strategy is adopted. For example, a prompt that integrates fact definition, ordinary user role, and local semantic granularity is constructed to generate general questions and answers for specific details; at the same time, a prompt that integrates relational reasoning, professional role, and global semantic granularity is constructed to generate in-depth questions and answers for the overall logic.

[0065] The large language model responds to these multi-perspective prompts, outputting a structured set of candidate question-answer pairs. Since different information types correspond to different levels and granularities of semantic content, multi-type question-answer generation effectively breaks the limitations of a single generation mode, significantly improving text information coverage and question-answer data diversity.

[0066] Based on the above embodiments, the process of generating supplementary question-and-answer pairs in step S130 will be described in detail below.

[0067] Based on the text semantic block and the candidate question-answer pair set, the remaining semantic information that is not covered is identified and supplementary question-answer pairs are generated. This includes: simultaneously inputting the text content of the text semantic block and the candidate question-answer pair set into the large language model; guiding the large language model to compare the text content of the text semantic block with the answer content in the candidate question-answer pair set through prompts to identify the remaining semantic information in the text semantic block that is not covered by the answer content; constructing targeted generation prompts based on the remaining semantic information, and calling the large language model to generate supplementary question-answer pairs for the remaining semantic information; wherein, the answers of the supplementary question-answer pairs are constrained to be restates or summaries of the remaining semantic information.

[0068] It is easy to understand that the candidate question-answer pair set generated in step S120 and the corresponding text semantic block are input into the large language model. By constructing specific comparison prompt words, the large language model is guided to perform semantic difference analysis.

[0069] Specifically, the comparison prompts require the model to deeply understand the complete semantic content of the text semantic block and break it down into several independent semantic units, such as key facts, logical arguments, and step descriptions. Then, it examines the answer content in the candidate question-answer pair set one by one to determine whether the semantic unit has been effectively covered by these answers, thereby filtering out the remaining semantic information that exists in the text semantic block but is not mentioned or is not mentioned sufficiently in all existing answers.

[0070] For example, if a text semantic block contains "photosynthesis requires light, water, and carbon dioxide, and releases oxygen," while the candidate question-answer pair only covers "What is photosynthesis?" and "Does photosynthesis require light?", then the remaining semantic information identified by the model is "The raw materials for photosynthesis include water and carbon dioxide" and "The products of photosynthesis include oxygen."

[0071] After identifying the remaining semantic information, a targeted prompt is constructed based on this information, and a large language model is invoked to generate supplementary question-answer pairs. The targeted prompt explicitly states, "Please generate a natural and fluent question and its corresponding answer based on the following uncovered semantic information: 'remaining semantic information content'." Furthermore, to prevent the model from generating illusions, the targeted prompt strictly sets the constraint that "the answer must be a restatement or summary of the above semantic information, and the introduction of knowledge or speculation outside the text's semantic blocks is strictly prohibited."

[0072] Taking photosynthesis as an example again, the model will generate the question "What are the main raw materials of photosynthesis?" in response to the statement "The raw materials include water and carbon dioxide", and generate the answer: "According to the original text, the main raw materials of photosynthesis include water and carbon dioxide". In this way, it ensures that each generation accurately fills in the information gap.

[0073] To further optimize output quality during the generation of supplementary question-and-answer pairs, multi-dimensional generation control strategies can be introduced. These strategies can be implemented by adjusting the prompt parameters. For example, in subsequent iterations, the perspective of information type or role setting can be actively switched to uncover remaining information at different levels. Alternatively, if the remaining information is mostly detailed, the model can be guided to focus on a global summary, and vice versa. Furthermore, for complex logic, the model can be prompted to perform multi-step reasoning to generate answers; for simple facts, the answers can be kept concise.

[0074] The newly generated supplementary question-answer pairs are added to the current candidate question-answer pair set, forming the updated set, i.e., the expanded question-answer pair set. At the same time, it is determined whether the current iteration meets the preset stopping condition. If it does not meet the condition, a new round of iteration is started; if it does meet the condition, the process is terminated and the final expanded question-answer pair set is output.

[0075] The preset stopping conditions include convergence stopping conditions, quantity limit stopping conditions, and round limit stopping conditions. Convergence stopping conditions occur when no new valid supplementary question-answer pairs are generated for several consecutive rounds (e.g., 2 or 3 rounds), indicating that most of the extractable information in the text semantic block has been captured, and the benefit of continuing iteration is extremely low. Quantity limit stopping conditions occur when the total number of supplementary question-answer pairs has reached a preset maximum limit (e.g., 20 pairs per text semantic block) to prevent data redundancy. Round limit stopping conditions occur when the number of iteration rounds has reached a preset maximum limit (e.g., 5 rounds), serving as a fallback to prevent infinite loops.

[0076] In this embodiment, based on the text semantic block and the candidate question-answer pair set, the remaining uncovered semantic information is identified and supplementary question-answer pairs are generated. This process is iteratively executed until a preset stopping condition is met, resulting in an expanded question-answer pair set. This method substantially improves the knowledge output density of a single text fragment by progressively mining the implicit information in the text semantic block until a preset coverage saturation is reached.

[0077] Based on the above embodiments, the following will further describe in detail the verification and deduplication process in step S140.

[0078] The extended question-and-answer pair set is subjected to validation and deduplication, including: performing semantic consistency validation on the supplementary question-and-answer pairs in the extended question-and-answer pair set to determine whether the question and answer match; performing answer verifiability validation on the supplementary question-and-answer pairs in the extended question-and-answer pair set to determine whether the answer can be verified from the semantic blocks of the text; and filtering out duplicate and similar question-and-answer pairs in the supplementary question-and-answer pairs in the extended question-and-answer pair set to remove supplementary question-and-answer pairs with semantic repetition.

[0079] When performing answer verifiability checks on supplementary question-answer pairs in the extended question-answer pair set, if it is determined that the answer contains information that cannot be directly supported or deduced from the text semantic block, the supplementary question-answer pair is judged as invalid and removed.

[0080] It is easy to understand that after obtaining the expanded question-answer pair set after multiple rounds of iterative completion, strict quality control needs to be performed. That is, by combining the reasoning ability of the large model with the rule matching method, each supplementary question-answer pair in the expanded question-answer pair set is subjected to multi-dimensional verification and deduplication to eliminate noisy question-answer pairs and ensure the high quality and reliability of the final output target question-answer pair dataset.

[0081] Specifically, firstly, a semantic consistency check is performed between the question and the answer to filter out samples that are irrelevant or logically disjointed. Each supplementary question-answer pair is input into the large language model, and a judgment instruction is issued, requiring the model to determine whether the answer accurately addresses the core requirement of the question based on semantic logic. For example, if the question is "What are the raw materials for photosynthesis?", and the answer is "Plants absorb sunlight through their leaves," the model will determine that although the answer is related to the topic, it does not hit the core element of the raw materials that the question focuses on, thus marking it as semantically inconsistent and removing it.

[0082] Secondly, verifiability checks are performed on the answers to prevent the large model from creating illusions. The answer text in the supplementary question-and-answer pair is aligned and compared with the context of the original text semantic block. If it is determined that the answer contains information that cannot be directly supported or derived from the text semantic block, such as the answer mentioning numbers, entities, dates, or subjectively assumed causal relationships not present in the original text, then the supplementary question-and-answer pair is determined to be invalid. For example, if the text semantic block only mentions "a certain drug is effective," and the answer claims "the cure rate of this drug is 99%," the question-and-answer pair will be determined to be unverifiable and eliminated due to the lack of support from the original text. In particular, low-information questions and answers such as "What does Newton's First Law say?" or "It says some knowledge," which are semantically consistent but extremely empty in information and lack substantial knowledge contribution, are also identified and eliminated in this embodiment.

[0083] Subsequently, duplicate or highly similar question-and-answer filtering is performed. Given that multi-perspective generation and iterative completion may lead to semantic overlap, this embodiment calculates the semantic similarity between all question-and-answer pairs in the expanded question-and-answer pair set. Specifically, a text embedding model can be used to encode questions and answers into high-dimensional vectors, and the cosine similarity between the vectors can be calculated. If the similarity between two sets of questions and answers exceeds a preset threshold (e.g., 0.85), they are determined to be duplicates or highly similar. In this case, the information completeness of the two sets is compared, retaining only the winner with more standardized expression and more comprehensive information, while redundant items are removed, thus ensuring the diversity of the dataset.

[0084] In addition, the verification process can be further included in semantic coverage validity verification and question-answer independence verification. The former assesses whether a single question-answer pair anchors to key information points in the original text, preventing the generation of marginal or meaningless questions and answers; the latter determines whether the question-answer pair has independent semantic integrity, eliminating fragmented questions and answers that require context to understand.

[0085] After the semantic consistency check, answer verifiability check, and duplicate and similar question-answer filtering processes, the expanded question-answer pair set has had its noisy data significantly removed. Finally, the denoised question-answer pairs are aggregated into the target question-answer pair dataset, which can be directly used for supervised fine-tuning training of large language models or imported into the knowledge base of retrieval enhancement generation systems.

[0086] also, Figure 2 This diagram illustrates the overall process of the question-answer pair generation method provided in this embodiment of the invention, showcasing the overall architecture of the intelligent question-answer pair generation, completion, and verification system based on a large language model. Figure 2 As shown, it mainly includes four core processing stages. The first stage is the multi-perspective question-and-answer generation stage. This stage approaches the problem from three dimensions: information type, role setting, and information granularity. Through multi-perspective prompts, it achieves preliminary mining of text semantic blocks and generates a basic set of candidate question-and-answer pairs.

[0087] Then there is the coverage enhancement and dynamic completion mechanism. The large language model analyzes the comprehensiveness of the current candidate question-answer set in covering the original text, identifies the remaining semantic information that has not been effectively covered, and generates supplementary question-answer pairs in a targeted manner. This process continues until a preset stopping condition is met, such as no new information for several consecutive rounds, so as to obtain an expanded question-answer pair set with more complete information.

[0088] The next step is the question-and-answer verification and screening mechanism, which includes verifying the semantic consistency between questions and answers, verifying whether the answers can be derived from the original text, filtering redundant questions and answers that are semantically repetitive or highly similar, and eliminating invalid or low-information question-and-answer pairs.

[0089] Finally, a high-quality target question-answer pair dataset, after layers of screening and optimization, is output and can be used for subsequent model training or knowledge base construction.

[0090] Corresponding to the question-answer pair generation methods described in the above embodiments, the present invention also provides a question-answer pair generation apparatus.

[0091] Figure 3 A schematic diagram of the question-and-answer pair generation device provided in an embodiment of the present invention is shown. Figure 3As shown, the device includes: a text semantic block acquisition module 310, used to acquire text semantic blocks to be processed; a candidate question-answer pair generation module 320, used to generate a set of candidate question-answer pairs corresponding to the text semantic blocks based on multi-perspective prompts; the multi-perspective prompts include at least two of the following: information type, role setting, and information granularity; a supplementary question-answer pair generation module 330, used to identify uncovered remaining semantic information and generate supplementary question-answer pairs based on the text semantic blocks and the set of candidate question-answer pairs, thereby obtaining an expanded question-answer pair set; and a target question-answer pair output module 340, used to perform verification and deduplication processing on the expanded question-answer pair set, and output a target question-answer pair dataset.

[0092] In this embodiment, the text semantic block acquisition module 310 acquires the text semantic block to be processed. The candidate question-answer pair generation module 320 generates a set of candidate question-answer pairs corresponding to the text semantic block based on multi-perspective prompts. The multi-perspective prompts include at least two of the following: information type, role setting, and information granularity. Then, the supplementary question-answer pair generation module 330 identifies the remaining uncovered semantic information based on the text semantic block and the candidate question-answer pair set and generates supplementary question-answer pairs to obtain an expanded question-answer pair set. Finally, the target question-answer pair output module 340 performs verification and deduplication processing on the expanded question-answer pair set and outputs the target question-answer pair dataset. This device, through multi-perspective prompt guidance and iterative completion of remaining semantic information, achieves full mining of multi-level knowledge within a single text semantic block, significantly improving the information coverage and diversity of question-answer pairs. At the same time, combined with answer verifiability and deduplication filtering, it effectively eliminates illusory content and redundant data, ensuring the high quality and reliability of the output question-answer pair dataset, which can be directly used for large language model training and knowledge base construction.

[0093] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a question-answer pair generation method, which includes: acquiring a text semantic block to be processed; generating a candidate question-answer pair set corresponding to the text semantic block based on multi-perspective prompts; the multi-perspective prompts include at least two of the following: information type, role setting, and information granularity; identifying uncovered remaining semantic information and generating supplementary question-answer pairs based on the text semantic block and the candidate question-answer pair set, obtaining an extended question-answer pair set; performing verification and deduplication processing on the extended question-answer pair set, and outputting a target question-answer pair dataset.

[0094] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0095] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the question-answer pair generation method provided by the above methods. The method includes: acquiring a text semantic block to be processed; generating a candidate question-answer pair set corresponding to the text semantic block based on multi-perspective prompts; the multi-perspective prompts include at least two of information type, role setting, and information granularity; identifying uncovered remaining semantic information and generating supplementary question-answer pairs according to the text semantic block and the candidate question-answer pair set, thereby obtaining an extended question-answer pair set; performing verification and deduplication processing on the extended question-answer pair set, and outputting a target question-answer pair dataset.

[0096] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the question-answer pair generation method provided by the above methods. The method includes: acquiring a text semantic block to be processed; generating a candidate question-answer pair set corresponding to the text semantic block based on multi-perspective prompts; the multi-perspective prompts include at least two of information type, role setting, and information granularity; identifying uncovered remaining semantic information and generating supplementary question-answer pairs according to the text semantic block and the candidate question-answer pair set, thereby obtaining an extended question-answer pair set; performing verification and deduplication processing on the extended question-answer pair set, and outputting a target question-answer pair dataset.

[0097] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0098] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A question-answer pair generation method, characterized in that, include: Obtain the semantic block of the text to be processed; Based on multi-perspective prompts, a set of candidate question-answer pairs corresponding to the text semantic block is generated; The multi-perspective prompts include at least two of the following: information type, role setting, and information granularity. Based on the text semantic block and the candidate question-answer pair set, identify the remaining semantic information that has not been covered and generate supplementary question-answer pairs to obtain an extended question-answer pair set; The extended question-answer pair set is validated and deduplicated to output the target question-answer pair dataset.

2. The question-answer pair generation method according to claim 1, characterized in that, The information types include at least two of the following: fact definition type, relational reasoning type, and process summary type, used to cover entity attributes and concept interpretations in the text semantic block; and / or, The role settings include professional roles and ordinary user roles, used to cover causal logic, conditional dependencies, and contrastive differences in the semantic blocks of the text; and / or, The information granularity includes global semantic granularity and local semantic granularity, which are used to cover the process steps and overall theme in the text semantic block.

3. The question-answer pair generation method according to claim 1 or 2, characterized in that, The step of identifying uncovered remaining semantic information and generating supplementary question-answer pairs based on the text semantic block and the candidate question-answer pair set includes: The text content of the text semantic block and the candidate question-answer pair set are simultaneously input into the large language model. The large language model is guided by prompts to compare the text content of the text semantic block with the answer content in the candidate question-answer pair set, and to identify the remaining semantic information in the text semantic block that is not covered by the answer content. Based on the remaining semantic information, a targeted prompt is constructed, and a large language model is invoked to generate supplementary question-and-answer pairs for the remaining semantic information; wherein, the answers to the supplementary question-and-answer pairs are constrained to be restates or summaries of the remaining semantic information.

4. The question-answer pair generation method according to any one of claims 1-3, characterized in that, The step of identifying uncovered remaining semantic information and generating supplementary question-answer pairs based on the text semantic block and the candidate question-answer pair set is executed iteratively until a preset stopping condition is met, resulting in an expanded question-answer pair set; The preset stop condition includes any one of the following: No new valid supplementary question-and-answer pairs were generated in multiple consecutive rounds; The total number of supplementary question-and-answer pairs has reached the preset upper limit; The number of iterations has reached the preset maximum number of iterations.

5. The question-answer pair generation method according to any one of claims 1-4, characterized in that, The validation and deduplication process for the extended question-and-answer pair set includes: Semantic consistency checks are performed on the supplementary question-answer pairs in the extended question-answer pair set to determine whether the question and answer match. Perform answer verifiability verification on the supplementary question-answer pairs in the extended question-answer pair set to determine whether the answer can be verified from the text semantic block; Repeated and similar question-and-answer filtering is performed on the supplementary question-and-answer pairs in the extended question-and-answer pair set to remove supplementary question-and-answer pairs with semantic repetition.

6. The question-answer pair generation method according to claim 5, characterized in that, When performing answer verifiability verification on the supplementary question-answer pairs in the extended question-answer pair set, if it is determined that the answer contains information that cannot be directly supported or deduced from the text semantic block, the supplementary question-answer pair is determined to be an invalid question-answer pair and is removed.

7. The question-answer pair generation method according to any one of claims 1-6, characterized in that, The process of obtaining the text semantic block to be processed includes: The original text is semantically analyzed and segmented, and divided into at least one semantic block according to semantic integrity and length constraints.

8. A question-and-answer pair generation apparatus, characterized in that, include: The text semantic block acquisition module is used to acquire the text semantic blocks to be processed; The candidate question-answer pair generation module is used to generate a set of candidate question-answer pairs corresponding to the text semantic block based on multi-perspective prompts; the multi-perspective prompts include at least two of the following: information type, role setting, and information granularity. The supplementary question-answer pair generation module is used to identify the remaining semantic information that has not been covered based on the text semantic block and the candidate question-answer pair set, and generate supplementary question-answer pairs to obtain an extended question-answer pair set; The target question-answer pair output module is used to perform validation and deduplication processing on the extended question-answer pair set and output the target question-answer pair dataset.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the question-answer pair generation method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the question-answer pair generation method as described in any one of claims 1 to 7.

Images