A method and device for generating a question and answer pair dataset, and a question and answer model

Abstract

The application provides a method and device for generating a question and answer pair data set and a question and answer model, and belongs to the technical field of artificial intelligence. The method comprises: dividing each answer data source in an answer data set to obtain a plurality of first answer segments, each first answer segment being configured with source metadata; matching at least one first answer segment for each question in a question data set to construct a context set for generating an answer to the question; generating an answer for any question in the question data set based on the question and the context set of the question, constructing a question-based question and answer pair based on the question and the answer, the question and answer pair comprising the question, the answer to the question, and a reference mark of the answer; and taking the set of question and answer pairs corresponding to all questions as the question and answer pair data set. The application attaches source metadata to each answer segment obtained by dividing the answer data set, so that the answer to the question pair is configured with a clear reference mark, and the answer can be traced back to a specific data source.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus and question-answering model for generating question-answering datasets. Background Technology

[0002] The training performance of Large Language Models (LLMs) largely depends on the scale and quality of the training data. During the pre-training phase, the model builds its basic knowledge framework and language understanding capabilities by learning from massive amounts of text data. However, currently widely used pre-training datasets lack clear original source annotations; even question-and-answer pair datasets are difficult to trace back to specific data sources, causing Large Language Models to provide unverifiable answers. Summary of the Invention

[0003] This application provides a method, apparatus, and question-answering model for generating question-answering pairs datasets, aiming to solve the problem that high-quality datasets of large language models constructed by existing technologies lack clear original source annotations.

[0004] Firstly, this application provides a method for generating a question-answer pair dataset, comprising:

[0005] Each answer data source in the answer dataset is segmented to obtain multiple first answer fragments, each of which is configured with source metadata.

[0006] For each question in the question dataset, match at least one first answer fragment and construct a context set for generating the answer to the question;

[0007] For any question in the question dataset, an answer is generated based on the question and its context set. A question-answer pair is constructed based on the question and the answer, and the question-answer pair includes the question, the answer to the question, and the reference tag of the answer.

[0008] Construct a question-answer pair dataset based on the set of question-answer pairs corresponding to all questions.

[0009] As one example, after constructing question-and-answer pairs based on questions, the process also includes:

[0010] Conduct quality assessments on the question-and-answer pairs;

[0011] Furthermore, the set of all qualified question-answer pairs is used as the question-answer pair dataset.

[0012] As one example, after segmenting each answer data source in the answer dataset to obtain multiple first answer fragments, the process further includes:

[0013] Convert each first answer fragment into a first semantic vector;

[0014] An index vector library is built based on the first semantic vector and source metadata of all first answer fragments.

[0015] As an example, each question in the question dataset is configured with at least one answer source identifier, and each answer source identifier points to a first target answer data source in the answer dataset.

[0016] As an example, for each question in the question dataset, at least one first answer fragment is matched, and a context set for generating answers to the questions is constructed, specifically including:

[0017] Transform the problem into a second semantic vector;

[0018] The search scope for a question is determined in the index vector library based on the source identifier of the question's answer.

[0019] Within the search scope, retrieve a first preset number of candidate answer fragments that have the highest matching degree with the second semantic vector, and construct a candidate answer fragment set;

[0020] A context set for generating answers to questions is constructed based on a set of candidate answer fragments.

[0021] As one example, after obtaining the candidate answer fragment set, the process further includes:

[0022] Select the preferred answer fragment set from the candidate answer fragment set based on semantic similarity;

[0023] Furthermore, a context set for generating answers to questions is constructed based on the set of preferred answer fragments.

[0024] As one example, a context set for generating answers to questions is constructed based on a set of preferred answer fragments, specifically including:

[0025] Configure a unique reference tag for each preferred answer fragment in the preferred answer fragment set;

[0026] Combine each preferred answer fragment and its reference tag into a reference pair;

[0027] The set of reference pairs for all preferred answer fragments is used as the context set for the question.

[0028] As an example, obtaining the problem dataset specifically includes:

[0029] Extract the first key information of each source question in the source question dataset. The first key information is information related to the answer data source. The first key information includes the main content of the source question or the basic information of the answer data source.

[0030] Match the first key information with the second key information of each answer data source in the answer dataset;

[0031] If a second key piece of information exists that matches the first key piece of information, then the source question is treated as a question in the question dataset, and the source metadata corresponding to the answer data source to which the second key piece of information belongs is used as the answer source identifier of the source question.

[0032] As an example, extracting the first key information of the source problem specifically includes:

[0033] Determine the correlation between the main content of the source question and the data source of the second target answer;

[0034] If the relevance is higher than the second threshold, the basic information of the second target answer data source will be used as part of the first key information of the source question.

[0035] As one example, obtaining the problem dataset also includes:

[0036] When the distance between a question in the question dataset and any answer data source is greater than a third threshold, the number of questions corresponding to any answer data source is expanded, and the set of all expanded questions and the set of all source questions are combined into a question dataset.

[0037] As one example, expanding the number of questions corresponding to any answer data source specifically includes:

[0038] Determine the augmentation amount for any answer data source;

[0039] Extract at least one seed fragment from all first answer fragments from any answer data source;

[0040] Generate at least one first augmentation problem for each seed fragment.

[0041] As one embodiment, after extracting at least one seed fragment, the process further includes:

[0042] Extract a set of reference questions from the source questions corresponding to any answer data source;

[0043] Using the reference problem set as a reference, generate at least one first augmentation problem for each seed fragment.

[0044] As one embodiment, after generating at least one first augmentation problem for each seed fragment, the process further includes:

[0045] For each first extension problem, a quality screening is performed to obtain at least one second extension problem;

[0046] Add source metadata and source for each second augmentation problem, where the source indicates how the second augmentation problem was obtained through the augmentation pathway.

[0047] As one example, generating answers based on questions and their context sets, and constructing question-answer pairs based on questions and answers, specifically includes:

[0048] Construct prompt words based on the question and its context;

[0049] Add constraint instructions to the prompt words;

[0050] Construct the output format;

[0051] Input the prompt words and output format into the large language model to obtain the answer to the question generated by the large language model. The answer is configured with a reference mark.

[0052] Question-answer pairs are generated based on the question, the answer to the question, and the reference tag of the answer.

[0053] As one example, selecting a set of preferred answer fragments from a set of candidate answer fragments based on semantic similarity specifically includes:

[0054] Determine the first similarity between each candidate answer fragment and the second semantic vector;

[0055] Determine the second similarity between each candidate answer fragment and any other candidate answer fragment in the candidate answer fragment set;

[0056] The score of each candidate answer segment is determined by the weighted sum of the first similarity and all second similarities.

[0057] The set of candidate answer fragments with the highest scores in the second preset number is used as the preferred answer fragment set.

[0058] As one example, the quality assessment of question-answer pairs specifically includes:

[0059] Determine the validity of the citations of answers in question-answer pairs;

[0060] Determine the length of the answer in the question-answer pair;

[0061] If both the validity and length of the citation are within the corresponding preset range, the quality assessment result of the question-and-answer pair is determined to be qualified.

[0062] As one example, determining the citation validity of answers in a question-answer pair specifically includes:

[0063] Divide the answer into multiple second answer segments according to the citation markers in the answer;

[0064] Determine the semantic similarity between each second answer fragment and the first answer fragment referenced by the second answer fragment;

[0065] Determine the literal overlap between each second answer fragment and the first answer fragment referenced by the second answer fragment;

[0066] Using normalized weights, the semantic similarity and word overlap of each second answer fragment are weighted and summed to obtain the citation validity of the second answer fragment;

[0067] The citation validity of the answer is determined based on the citation validity of all second answer fragments in the answer.

[0068] Secondly, this application provides a device for generating a question-answer pair dataset, including a segmentation module, a context set acquisition module, a question-answer pair acquisition module, and a dataset acquisition module;

[0069] The segmentation module is used to segment each answer data source in the answer dataset to obtain multiple first answer fragments, each of which is configured with source metadata;

[0070] The context set acquisition module is used to match at least one first answer fragment for each question in the question dataset and construct a context set for generating the answer to the question;

[0071] The question-answer pair acquisition module is used to generate an answer for any question in the question dataset based on the question and the question's context set, and to construct a question-answer pair based on the question and the answer. The question-answer pair includes the question, the answer to the question, and the reference tag of the answer.

[0072] The dataset acquisition module is used to construct a question-answer pair dataset based on the set of question-answer pairs corresponding to all questions.

[0073] Thirdly, this application also provides a question-answering model, which is trained on a question-answering pair dataset obtained based on any of the above-mentioned question-answering pair dataset generation methods.

[0074] Fourthly, this application also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-described methods for generating question-and-answer pair datasets.

[0075] Fifthly, this application also provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the above-described methods for generating question-and-answer pair datasets.

[0076] Sixthly, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements any of the above-described methods for generating question-and-answer pair datasets.

[0077] The fundamental contribution of this application lies in providing a scalable, automated, and quality-controlled methodology for building a 'trustworthy AI' data infrastructure. It not only generates data but, more importantly, embeds a 'trust chain' (through reference tagging) at the data level. This makes the responses of AI models trained on this dataset inherently interpretable and verifiable, which is crucial for promoting the responsible application of large language models in high-risk fields such as finance, healthcare, and law. Attached Figure Description

[0078] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0079] Figure 1 This is one of the flowcharts illustrating the method for generating the question-answer pair dataset provided in this application;

[0080] Figure 2 This is the second flowchart illustrating the method for generating the question-answer pair dataset provided in this application;

[0081] Figure 3 This is one of the flowcharts illustrating the process of obtaining the problem dataset provided in this application;

[0082] Figure 4 This is one example of the prompt words provided in this application;

[0083] Figure 5 This is the second example of the prompt words provided in this application;

[0084] Figure 6 This is the third example of the prompt words provided in this application;

[0085] Figure 7 This is one of the flowcharts illustrating the process of obtaining the context set of the problem provided in this application;

[0086] Figure 8 This is the fourth example of the prompt words provided in this application;

[0087] Figure 9 This is the fifth example of the prompt words provided in this application;

[0088] Figure 10 This is one of the structural schematic diagrams of the question-answer pair dataset generation device provided in this application;

[0089] Figure 11 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0090] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0091] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. The terms "upper," "lower," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are used only for the convenience of describing this application and for simplifying the description, and do not indicate or imply that the device or element referred to must be configured in a specific orientation, constructed or operated in a specific orientation, and therefore should not be construed as a limitation of this application. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly, for example, as a fixed connection, a detachable connection, or an integral connection; a mechanical connection or an electrical connection; a direct connection or an indirect connection through an intermediate medium; or a connection within two elements. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0092] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0093] The following is combined Figures 1 to 11 This application describes the method, apparatus, and question-answering model for generating question-answer pairs datasets.

[0094] It should be noted that the question-answer pair dataset generation method provided in this application embodiment is implemented based on the question-answer pair dataset generation device. The question-answer pair dataset generation method enables each segment in the answer of the question-answer pair to be configured with clear source metadata, so that the answer can be traced back to a specific data source. The large language model trained on such a dataset can provide real answers when answering, avoiding model illusion.

[0095] This application describes the method for generating question-answer pair datasets using a question-answer pair dataset generation device as an example.

[0096] Figure 1 This is one of the flowcharts illustrating the method for generating the question-answer pair dataset provided in this application. Figure 2 This is the second flowchart illustrating the method for generating the question-and-answer pair dataset provided in this application.

[0097] like Figure 1 and 2 As shown, the method for generating the question-and-answer pair dataset provided in this application includes:

[0098] S110: Answer dataset Each answer data source in the process is segmented to obtain multiple first answer fragments, and each first answer fragment is configured with source metadata.

[0099] In step S110, while segmenting the answer data source (e.g., a book), source metadata is attached to the segmented answer fragments, providing a data foundation for providing answer references when subsequently constructing question-answer pairs. The answer fragments obtained from segmenting all answer data sources in the answer dataset are compiled into a fragment library. Please refer to [reference needed]. Figure 2 .

[0100] For each answer data source, it is segmented according to its own structural characteristics to obtain multiple answer fragments while ensuring semantic integrity.

[0101] The pre-training data for the currently widely used large language models mainly comes from general web crawling. Although the amount is huge, there are generally problems such as uneven content quality and lack of information accuracy. This makes the large language models more prone to hallucination when answering questions, that is, providing information that seems reasonable but is actually wrong or unverifiable.

[0102] Based on these considerations, in one possible implementation, each answer data source in the answer dataset is text with a clear structure or objective facts, without introducing subjective commentary data. The clear structure helps to accurately determine the source of the answer, ensuring that the question-answer pairs are built based on reliable data and avoiding subjective biases or unfounded statements from entering the training dataset.

[0103] In one possible implementation, the answer data source is structured book content (such as chapter text in Markdown format).

[0104] S120: For the problem dataset Each question in the algorithm matches at least one first answer fragment, and a context set is constructed to generate the answer to the question.

[0105] Please combine Figure 2 In this step, answer fragments from a fragment library are matched with each question to establish a correlation between questions and answers, providing a data foundation for the accuracy of question-answer pairs. In this step, the answer fragments matched for all questions are combined into an answer source fragment library.

[0106] In one possible implementation, each question in the question dataset comes from real community question data, which is more objective than comment data.

[0107] S130: For any question in the question dataset, generate an answer based on the question and the question's context set, and construct a question-answer pair based on the question and the answer. The question-answer pair includes the question, the answer to the question, and the reference tag of the answer.

[0108] In step S130, the question-answer pair includes a reference tag for the answer. Based on the reference tag, the source of the answer can be obtained, ensuring that the answer configuration has a clear source and improving the credibility of the model output. Furthermore, during the training process, the answer in the question-answer pair can be compared with the specific answer source based on the reference tag to determine the accuracy of the answer and reduce the model's illusion output.

[0109] S140: Construct a question-answer pair dataset based on the set of question-answer pairs corresponding to all questions. Please refer to Figure 2 .

[0110] In this application embodiment, each answer fragment obtained by segmenting the answer dataset is accompanied by source metadata, thereby configuring clear reference marks in the answers of question pairs, so that the answers can be traced back to specific data sources, and the answers output by the large language model have credibility and accuracy.

[0111] In step S110, the answer dataset is processed. The process of segmenting each answer data source in the database can be formalized as follows:

[0112] (1);

[0113] in, This is a fragment library obtained after segmenting the answer dataset.

[0114] The following explanation uses books as the source of answers as an example. The process of segmenting the answer source includes the following steps:

[0115] S1101: Perform a coarse segmentation of the answer data source content based on the built-in structured tags of the answer data source.

[0116] Given a dataset of real books ,in For the first This book. First, it uses the document's structural information for initial segmentation. For example, for Markdown format books, it detects the identifiers of each level of heading (e.g., Heading 1 (11), Heading 2 (1111), etc.). Based on these identifiers, the document is then divided into logical units by chapters / sections.

[0117] S1102: For each logical unit obtained from the initial segmentation, the logical unit is further subdivided based on the length standard to obtain smaller, semantically complete text fragments, forming answer fragments.

[0118] In this step, the length of each logical unit obtained from the initial segmentation is checked. The length of the logical unit is typically identified based on the number of tokens (which can be calculated using the tokenizer in the embedding model) or the number of characters. If the length of the logical unit exceeds a preset length threshold (determined based on retrieval accuracy and contextual completeness), a segmentation mechanism is triggered. If the length of the logical unit does not exceed the preset length threshold, then the logical unit is considered a first answer fragment.

[0119] The subdivision process is as follows: Based on the line break or sentence end punctuation mark (such as ".", "!") of the logical unit, the logical unit is divided into semantically complete sub-units. The core idea is to keep the sentence complete and not truncate it in the middle of the sentence. Usually, the blank line between paragraphs or after the period or line break is selected to divide the sentence and obtain multiple first answer segments.

[0120] S1103: Assign and store a set of key metadata for each final generated first answer fragment as source metadata for the answer fragment, so that the source can be referenced when generating the answer.

[0121] For books, the key metadata for each answer segment includes:

[0122] The ID (identifier) ​​of the book from which the answer fragment originates: clearly identifies the book to which the answer fragment belongs;

[0123] Chapter ID and Section ID: These specify the chapter and section to which the answer snippet belongs;

[0124] Unique Segment ID: Used to uniquely identify the answer segment. Generally, the segment ID can be composed of the book ID, chapter ID, section ID, and the sequential number of the answer segment in the original section, containing the hierarchical structure information of the answer segment's location;

[0125] Original offset information: The starting and ending characters, or word positions, of the answer fragment in the original document, which facilitates tracing and verification.

[0126] S1104: Construct a fragment library based on all first-answer fragments and their source metadata. .

[0127] All answer snippets and their source metadata will be collected into a snippet library. ( Indicates the number of fragments in the fragment library (The first answer fragment) is provided for later use. This book The fragment library can be represented as:

[0128] (2);

[0129] in, Indicates the first i This book The k A fragment of an answer.

[0130] This application embodiment systematically breaks down lengthy answer data sources into clearly structured answer fragments. It preserves the hierarchical information of the answer data source through source metadata and controls the fragment length, providing a foundation for subsequent matching of questions and answer fragments and generation of question-answer pairs.

[0131] In one possible implementation, after segmenting each answer data source in the answer dataset to obtain multiple first answer fragments (S110), the process further includes building an index vector library, specifically including:

[0132] P1: Convert each first answer fragment into a first semantic vector. (This refers to the fragment library.) All answer fragments are efficiently converted into high-quality, fixed-dimensional embedding vectors that are robust to semantic understanding and length variations.

[0133] In one possible implementation, an embedding model is used. Each first answer fragment is converted into a first semantic vector, preferably using the BGE-small model for the fragment library. Each answer fragment Encode it and map it to A dimensional vector space.

[0134] The key advantage of choosing the BGE-small model lies in its superior semantic representation capabilities and robustness to variations in text length, namely:

[0135] High-quality semantic representation: By training on massive and diverse corpora, the BGE-small model can capture the deep semantic features of text, not just the surface words. This means that even if two text fragments differ greatly in word usage, as long as their core semantics or intentions are similar, the vectors generated by the model will be very close in position in the vector space.

[0136] Robustness to Length Differences: BGE-small effectively measures semantic similarity even when dealing with text pairs of significantly different lengths. For example, BGE-small can accurately identify the semantic relationship between a short, concise question (as the question) and a longer, richer paragraph containing the answer, even if their lengths differ by several times, and generate vector representations with high similarity. This is crucial for question-answering scenarios, as it ensures that the question can accurately retrieve the longer text context containing the answer.

[0137] Furthermore, selecting an embedding model that performs well in downstream tasks such as question-answer pair generation (BGE model performs well on authoritative leaderboards such as MTEB (Massive Text Embedding Benchmark)) ensures that the 'similarity' measured in the retrieval stage (upstream task) is consistent with the 'semantic relevance' required to generate the final answer in the model's understanding, thereby maximizing the effectiveness of retrieval.

[0138] The vectorization process of the answer fragment based on the BGE-small model can be formalized as follows:

[0139] (3);

[0140] in, This is a fragment of the answer. The dense semantic vector, i.e. the first semantic vector.

[0141] P2: Construct an index vector library based on the first semantic vector and source metadata of all first answer fragments. Index vector library Maintain the mapping relationship between the first semantic vector and its source metadata. Index vector library. Vector indexing enables fast, large-scale similarity retrieval, providing technical support for subsequent steps. Please see the following explanation of step S120.

[0142] This application's embodiments, through the embedding model and efficient vector database, ensure high-quality semantic similarity search in a large-scale fragment library with sub-second response speed, laying a solid foundation for subsequent real-time and accurate question matching and large-scale question-answer pair generation, and guaranteeing scalability.

[0143] In one possible implementation, in step S120, each question in the question dataset is configured with at least one answer source identifier, and each answer source identifier points to a first target answer data source in the answer dataset.

[0144] In this embodiment of the application, the answer source identifier helps to limit the matching range to the answer fragment range of the first target answer data source pointed to by the answer source identifier when matching the first answer fragment for the question, thereby improving the matching efficiency.

[0145] Specifically, such as Figure 3 As shown, the problem dataset is obtained, specifically including:

[0146] S310: Extract the first key information for each source question in the source question dataset. The first key information is information related to the second target answer data source. The first key information includes the main content of the source question (plot, characters, settings, concepts, themes, etc.) or the basic information of the second target answer data source (e.g., book title, author, etc.). The second target answer data source is the answer data source obtained based on the main content of the source question.

[0147] The source dataset is a dataset obtained from a source of questions. For example, the community mentioned above is a source of questions, and the collection of community questions obtained from the community is the source dataset. The source questions in the source dataset are unfiltered questions.

[0148] S320: Match the first key information with the second key information of each answer data source in the answer dataset.

[0149] The second key information of the answer data source in the answer dataset includes the data source identity (e.g., book title, book ID), the source of the data source (e.g., book publication information), the author of the data source, the core content of the data source (e.g., book summary, keywords, main idea, theme, key plot, core concepts, main characters, etc.), and the topic classification of the data source.

[0150] S330: If there is a second key piece of information (there may be one or more) that matches the first key piece of information, then the source question is treated as a question in the question dataset, and the source metadata corresponding to the answer data source to which the second key piece of information belongs is used as the answer source identifier of the source question. All the selected source questions are combined to form the question dataset.

[0151] In this embodiment, the key information of the source question is matched with the key information of the answer data source to obtain the association between the source question and the answer data source. This association is then used as the answer source identifier for the source question, thereby filtering out questions related to the answer dataset, reducing questions unrelated to the answer dataset, and improving the accuracy of answer citation.

[0152] In one possible implementation, extracting the first key information of the source question specifically includes: determining the relevance between the main content of the source question and the second target answer data source. If the relevance is higher than a second threshold, then the basic information of the second target answer data source (such as book title, author, etc.) is included as part of the first key information of the source question.

[0153] This application embodiment determines the relevance of a question by comparing the main content of the source question with the content of the second target answer data source, thereby identifying accurate first key information. This filters out questions unrelated to the answer data source in the answer dataset (such as book derivatives, book version comparisons, book purchases, purchase channels, and reader subjective reviews), questions that the answer data source cannot answer (such as fan fiction based on books), and questions related to the answer data source but for which the answer data source itself cannot provide sufficient answers (such as in-depth speculations beyond the scope explicitly stated in the book, secondary creations based on the book's background, or subjective fantasies). It retains those questions that are highly relevant to the content of the answer data source and suitable for generating question-answer pairs, ensuring that the questions in the question dataset are "answerable" based on the answer data source.

[0154] Based on the above source question screening process, in one possible implementation, in step S310, the contextual understanding and reasoning capabilities of the Large Language Model (LLM) are utilized to achieve deep semantic understanding, relevance judgment, and intelligent matching between questions and books, thereby determining whether there is a substantial connection between the key information of the question and the answer data source and obtaining the first key information.

[0155] Specifically, for each source question, question prompts are first constructed to guide LLM analysis of the source question text, identifying whether the source question explicitly or implicitly points to a second target answer data source, such as content relevance judgment, book matching, or non-content question identification (i.e., the aforementioned relevance). Furthermore, the output format is constructed within the prompts. Figure 4 An example of a prompt word is shown.

[0156] Subsequently, the prompt words and output format are input into the LLM, enabling the LLM to understand and reason about the source problem and output the first key information for each source problem.

[0157] Taking books as the source data for the answers, the above process of selecting source questions can be formalized as follows:

[0158] (4);

[0159] in Construct a constructor for the prompt words. For the source problem, , These are the book title and author name from the source data of the second target answer, which are the best matches based on the question prompts, as output by the LLM model. The model's matching factors include content relevance, the best-matching book title and its author name (relevant and matching author names), etc. In addition, the model also outputs content relevance (indicating whether the question is related to the book discussed in the question) and a label indicating whether it is a non-content question, allowing the LLM to make layer-by-layer judgments, reducing illusions and improving judgment accuracy.

[0160] Based on the above LLM implementation, after obtaining the book name and author name identified by LLM, in step S320, the book name and author name are fuzzily matched with the book name and author name of the answer data source in the answer dataset.

[0161] In one possible implementation, Levenshtein distance is used to measure the similarity between strings, achieving fuzzy matching. The function that calculates this distance is denoted as LD(str1, str2), and its return value represents the minimum number of single-character editing operations (insertion, deletion, or replacement) required to transform string str1 into string str2. To obtain a standardized similarity score, the book names of the source questions identified by LLM are calculated using the following formula. (i.e., the first key information) and the book title of the answer data source String similarity between (i.e., the second key information) :

[0162] (5);

[0163] in This represents the length of the string.

[0164] Similarly, the author names of the identified source problems can be calculated. (i.e., the first key information) and the author's name in the answer data source of the answer dataset. String similarity between (i.e., the second key information) .

[0165] In step S330, between the source question and an answer data source in the answer dataset, if the string similarity... and If all values ​​are above the threshold, the source question passes the screening and is included as a question in the question dataset. The answer data source is designated as the first target answer data source for the source data, and the source metadata of the first target answer data source is used as the answer source identifier for the source question.

[0166] Finally, the collection of all filtered source data (with answer source identifiers) As a problem dataset, in which, (6);

[0167] in, To filter the obtained source questions, ( ) for and The associated book IDs. Therefore, these source questions have been preliminarily determined to be related to the answer data sources in the answer dataset, and can be answered based on the content of these source data sources. Furthermore, these questions are not derivative or subjective questions, laying a solid foundation for the generation of high-quality question-answer pairs in the future.

[0168] This application employs a two-stage strategy: first, LLM identifies key information of the source question, and then fuzzy matching is used for verification. This decouples the expensive LLM inference task (identification) from the efficient traditional algorithm task (pattern matching), avoiding the need to input both the question dataset and the answer dataset into the LLM to match the answer data source for the source question. Inputting both datasets would not only greatly increase the single processing cost and lexical consumption of the LLM, but may also affect the model's judgment accuracy due to excessively long context. Therefore, the two-stage strategy of this application significantly improves the efficiency and scalability when processing large-scale question datasets.

[0169] The demand for high-quality data is surging across various fields, but it is becoming increasingly difficult and costly to obtain high-quality, diverse raw text data (such as professional books and in-depth community discussions) directly from the internet on a large scale.

[0170] Based on these considerations, one possible implementation of obtaining the problem dataset also includes:

[0171] If the distance between a question in the question dataset and any answer data source is greater than a third threshold, the number of questions corresponding to that answer data source is expanded, and the set of all expanded questions is increased. and the set of all source problems Combined into a problem dataset : .

[0172] Each expansion question is configured with an answer source identifier, which points to a first target answer data source in the answer dataset, i.e., the answer data source that the expansion question targets.

[0173] In one possible implementation, the augmented question is also configured with a similarity identifier that indicates the initial similarity between the augmented question and the corresponding first target answer data source.

[0174] Based on high-quality source questions, this application generates diverse and high-quality new questions for answer data sources that have not been sufficiently questioned or whose question coverage is insufficient. This expands the content coverage of the question data. The expanded overall question dataset serves as the starting point for subsequent question-answer pair construction, significantly improving the breadth and depth of the final generated question dataset's coverage of the answer dataset. Furthermore, by expanding the questions, the dependence on acquiring new original data can be significantly reduced.

[0175] In one possible implementation, the number of questions corresponding to any answer data source is expanded, specifically including:

[0176] Q1: Determine the augmentation of any answer data source.

[0177] Specifically, based on the question distance of the source question corresponding to each answer data source in the answer dataset. With preset parameters (i.e., the third threshold) determines the expansion amount.

[0178] Taking books as the source of answers as an example, for books and the book's excerpt library The definition is as follows:

[0179] (7);

[0180] in Is with Related source problem dataset The number of source problems, Reflects the source problem dataset The average number of readers who read this data fragment library The distance between the source question and the source question is the number of answer fragments in the dataset that will lead to a source question in the dataset.

[0181] Introduce preset parameters This parameter represents the desired number of books. The average readership of the source problem dataset is [number] people who have read this book. How many answer fragments in the fragment library are needed to pose a question? By comparison... and To determine the books Expanding on the question of whether and how many books are needed, note the relevant information. The set of extended problems is Then the size of the set should be:

[0182] (8);

[0183] This formula ensures that when the question distance of the source questions in a book is lower than or equal to a preset parameter (i.e., the number of source questions is too high), the number of expansion questions is 0. When the question distance of the source questions is greater than the preset parameter (i.e., the number of source questions is insufficient), the number of expansion questions is determined by the difference between the preset parameter and the number of source questions, and the two are directly proportional. Specifically, when the number of source questions is 0, i.e. Sometimes, At this point, the number of expanded questions is the expected total number of questions the book should have. This formula will... and The "distance" between them is linked, aiming to increase the total number of questions in the book to a desired level by expanding the questions. The technical idea behind this formula is to "align" the question coverage density of all answer data sources to a preset desired level through intelligent expansion, thereby ensuring that the final generated dataset is more evenly distributed across different knowledge topics and avoiding model bias caused by uneven distribution of the original question data.

[0184] Q2: Extract at least one seed fragment from all first answer fragments of any answer data source.

[0185] For each answer data source requiring question expansion, a predetermined number N answer fragments are randomly selected as seed fragments from the corresponding "fragment library" according to a certain strategy. The sampling strategy may consider the fragment length, information density, or importance in the structure of the answer data source (e.g., prioritizing extraction from core sections) in order to generate more valuable expansion questions.

[0186] Q3: Generate at least one first augmentation problem for each seed fragment.

[0187] In one possible implementation, for each seed fragment, by providing the text content of that seed fragment, LLM is used to generate one or more factual questions that can be clearly answered by that seed fragment, such as example prompts. Figure 5 As shown.

[0188] This application's embodiments, based on the segmentation of the answer data source, determine seed segments for question expansion by combining the data distribution of source questions within the answer data source. This systematically generates diverse and user-friendly questions for the insufficiently covered parts of the answer data source, thereby effectively improving the coverage depth of the final dataset to existing knowledge sources. Furthermore, the use of models for intelligent expansion can meet the data volume requirements of large language models.

[0189] In one possible implementation, after extracting at least one seed fragment, the process further includes:

[0190] Extract a reference question set from the source questions corresponding to any answer data source, and use the reference question set as a reference to generate at least one first expansion question for each seed fragment.

[0191] One possible implementation uses a probabilistic sampling function. From the problem dataset of the source problem The problem of sampling a predetermined number (e.g., 3) with equal probability without replacement. Sampling function. The definition is as follows:

[0192] (9);

[0193] in It is a random number taken from a uniform distribution. This represents the problem dataset from the source problem. The problem involves sampling three sources with equal probability without replacement.

[0194] Specifically, a probability p (e.g., p=0.8) is set, representing the proportion of augmented problems generated by the reference source problem among all augmented problems. Before calling LLM to generate problems for a seed fragment, a decision is made based on probability p whether to introduce a "reference problem set". If it is decided to introduce a reference, the problem dataset from the source problem is used. A predetermined number of source problems are randomly selected without replacement to form a "reference problem set". The LLM's prompts (when introducing references) are adjusted, such as... Figure 6 As shown. Relative to Figure 5 The prompt word includes a reference question as an example.

[0195] In this embodiment of the application, by randomly combining source question samples as part of the context, the aim is to utilize the natural language distribution characteristics of the source questions to "perturb" and guide the LLM, thereby improving the diversity of generated questions and preventing the model from falling into an inherent generation pattern.

[0196] In one possible implementation, after generating at least one first augmentation problem for each seed fragment, the following is also included:

[0197] Each first expansion question undergoes quality screening to obtain at least one second expansion question. Quality screening can filter out questions that are highly opinionated or have excessively open-ended answers.

[0198] Add source metadata and source for each second augmentation problem, indicating that the second augmentation problem was obtained through an augmentation pathway (e.g., "LLM generation-augmentation").

[0199] This application embodiment ensures that the expanded questions are factual and that, theoretically, the answers can be found in their associated seed fragments by performing quality screening on the expanded questions. Expanding the questions not only effectively increases the coverage of book content but also improves the quality and diversity of generated questions by introducing real-world data characteristics, providing rich and high-quality question resources for subsequently building comprehensive and high-quality question-answer pairs.

[0200] In step S120, as Figure 7 The image shows the problem dataset. Each question in Match at least one first answer fragment to construct a context set for generating the answer to the question. Specifically, it includes:

[0201] S710: Transform the problem into a second semantic vector.

[0202] In one possible implementation, an embedding model is used. The problem This is transformed into a second semantic vector. In this step, the input data for the embedding model is the problem dataset. The problem and its corresponding answer source identifier ( The output data is the second semantic vector. .

[0203] S720: Question-based answer source identification In the index vector library Determine the problem The search scope is such that subsequent searches are limited to the answer source identifier. The search is performed within the primary target answer data source, thus significantly reducing the number of vectors that need to be compared.

[0204] S730: Within the search scope, retrieve a first preset number of candidate answer fragments that have the highest matching degree with the second semantic vector, and construct a candidate answer fragment set. .

[0205] In one possible implementation, during the search, the computation problem... The cosine similarity between the first answer fragment and each first answer fragment within the search range is used to select the first answer fragments with the highest cosine similarity to a first preset number (K) of fragments as candidate answer fragments, thus forming a candidate answer fragment set. .

[0206] In one possible implementation, it is based on the FAISS index vector library. Approximate Nearest Neighbor (ANN) search algorithms (such as IVFADC, HNSW, etc.) are used to determine the candidate answer fragment set. Specifically, for a given second semantic vector The search algorithm will retrieve the top K most similar first semantic vectors from the indexed vector library. (One first semantic vector corresponds to one first answer fragment), thus forming a vector set of candidate answer fragments. ,Right now:

[0207] (10);

[0208] in This represents the first semantic vector with sort number k. This means selecting the K most similar values. These algorithms can significantly improve retrieval efficiency on large-scale datasets while sacrificing minimal precision.

[0209] The set of vectors of candidate answer fragments Corresponding candidate answer fragment set ， in From the index vector library The retrieved number Individual and Problem Similar candidate answer fragments.

[0210] S740: Construct a context set for generating answers to questions based on a set of candidate answer fragments. .

[0211] Finally, the context sets of all questions in the question dataset are aggregated to form a complete answer source fragment library. It can be formally represented as:

[0212] (11);

[0213] In this embodiment, by leveraging the indexing function of the index vector library, the search scope of the question is greatly reduced based on the answer source identifier of the question. Furthermore, the context set of the question is quickly obtained through the ANN search algorithm of the index vector library, which improves the efficiency of obtaining the context set. Moreover, it provides high-quality and organized factual evidence for the subsequent "citation-based answer generation" stage, which is a key foundation for achieving answer traceability.

[0214] In one possible implementation, after obtaining the set of candidate answer fragments, the following is also included:

[0215] Based on semantic similarity, from the candidate answer fragment set Selecting the best answer snippets Furthermore, based on the set of preferred answer fragments... Construct a context set to generate answers to questions.

[0216] In this embodiment, preferred answer fragments with high semantic similarity to the question are obtained from the candidate answer fragment set based on semantic similarity, thereby improving the matching between the question and the answer and improving the accuracy of the answer.

[0217] In one possible implementation, a set of preferred answer fragments is selected from the set of candidate answer fragments based on semantic similarity, specifically including:

[0218] R1: To determine the fragment of each candidate answer. With the second semantic vector The first similarity between them.

[0219] R2: To determine the fragment of each candidate answer. Compared with any other candidate answer fragment in the candidate answer fragment set The second similarity between them.

[0220] R3: Obtain the weighted sum of the first similarity and all second similarities for each candidate answer segment to determine the score of the candidate answer segment.

[0221] R4: Select the set of candidate answer fragments with the highest scores (the second preset number) as the preferred answer fragment set.

[0222] In one possible implementation, a maximum marginal relevance (MMR) optimization algorithm is used to select from the candidate answer fragment set. Selecting the best answer snippets The MMR (Mean Response Ratio) optimization algorithm can balance candidate answer fragments with the question. The algorithm considers the correlation between candidate answer fragments and the diversity among them, eliminates redundant information, and selects the fragment combination with the highest information gain. The input data for the MMR (Multi-Rate Optimization) algorithm includes a set of candidate answer fragments. The set of preferred answer fragments includes the number of preferred answer fragments and the balance factor. The output data is the set of preferred answer fragments. .

[0223] The formula for the MMR optimization algorithm is as follows:

[0224] (12);

[0225] in, The score for a given candidate answer segment. The first similarity score, For the second similarity, This represents the maximum value of the second similarity. (Based on the normalized coefficients...) The system aims to balance the relevance of candidate answer fragments to the question with the diversity among these fragments. A higher λ value tends to select the fragment most relevant to the question, potentially leading to redundancy. Conversely, a lower λ value prioritizes diversity, potentially sacrificing some relevance. In practice, the value of λ can be adjusted based on varying requirements for the completeness and conciseness of the answer.

[0226] In this embodiment, not only is the semantic similarity between candidate answer fragments and questions considered, but second candidate answer fragments that are highly similar to or repeat the content of the first candidate answer fragment are also penalized. This balances the contradiction between the relevance of answer fragments and questions and the complementarity between answer fragments, obtaining the text fragments that are most relevant to the questions and complementary in content. This ensures that the context of the answers in the final generated question-answer pair is both highly relevant and contains rich and non-redundant information.

[0227] In one possible implementation, step S740 involves constructing a context set for generating answers to the question based on the preferred answer fragment set, specifically including:

[0228] S7401: A collection of preferred answer snippets Each preferred answer fragment Configure a unique reference tag (e.g., [1], [2], [3]), and record the mapping relationship between the citation mark and the text of the corresponding preferred answer fragment. Each citation mark clearly corresponds to a first answer fragment that has been filtered and verified, laying a solid foundation for generating credible and traceable answers for subsequent question-and-answer pairs.

[0229] S7402: Extract each preferred answer segment And the reference mark of the preferred answer fragment Combined into a reference pair .

[0230] S7403: Submit all selected answer segments The set of reference pairs serves as the context set of the problem. .

[0231] In this application embodiment, a sufficient, diverse, high-quality context set with citation tags is matched for each issue to provide high-quality, organized factual evidence with clear source citations.

[0232] In step S130, in one possible implementation, a large language model (LLM) is used to generate question-answer pairs. This model is a large model with good instruction-following ability and contextual understanding of long texts. It takes both the question and the context set as input to generate answers that are accurate, logically clear, linguistically natural, and explicitly include source metadata. Step S130 can be abstractly represented as:

[0233] (13);

[0234] in, , , These are, respectively, a question dataset, a library of answer source fragments, and a set of question-answer pairs with citation tags.

[0235] Specifically, answers are generated based on questions and their context, and question-answer pairs are constructed based on questions and answers, including:

[0236] S1301: Construct prompt words based on the question and the context of the question.

[0237] Specifically, answer generation employs search-enhanced prompts. (Prompt word constructor) The problem The text, the context of the question Format the prompt according to a preset template to construct a structured prompt. For example, the context set All preferred answer snippets and its reference tags ( ( Clearly list and attach the questions. The prompts include the question itself and reference markers for each preferred answer segment, such as... Figure 8 As shown.

[0238] To ensure answer traceability, a source metadata mechanism is introduced during the generation process. Specifically, prompt words guide the large language model to insert a citation mark of the corresponding source fragment (i.e., the preferred answer fragment) after each key statement in each answer. For example, after the model answers a certain fact, a citation mark in the form of "[1]" is appended to the fact statement text, indicating that the fact stated in the text refers to the fragment that provides that fact (with a citation mark of "[1]"). Figure 8 It can be seen that the prompt contains explicit instructions, guiding... When generating answers, if the content of the statement comes from a fragment of a preferred answer. Then a corresponding quotation mark needs to be added at the end of the sentence. .

[0239] In one possible implementation, a sample format with citation marks can be included in the prompt to show the LLM the correct, citation-marked answer format and behavior pattern, requiring the LLM to answer based on the provided information and indicate the citation marks.

[0240] S1302: Add constraint instructions to the prompt words.

[0241] Specifically, the prompt word constructor By including restrictive instructions in the prompts (such as "must be based solely on the provided information" or "should not contain any external knowledge"), the likelihood of LLMs producing illusions or deviating from the information can be minimized.

[0242] S1303: Construct the output format.

[0243] When calling large model interfaces in batches, standardizing the output format of large models can facilitate subsequent data processing and greatly reduce the illusion content of the models.

[0244] In one possible implementation, the LLM is specified to output results in a structured format such as JSON. In addition to the generated answer, it should also include reference tags for the answer fragments referenced by the answer, so as to facilitate subsequent processing and traceability. Figure 9 An example of a prompt word in the output format is shown.

[0245] S1304: Input the prompt words and output format into the large language model to obtain the answer to the question generated by the large language model. ,Answer This includes the complete answer text (containing multiple second answer fragments) and reference tags for the second answer fragments. This process can be formalized as:

[0246] (14);

[0247] For example, the answer is “According to the data, the technology was first proposed in 1990[1] and subsequently applied[2].” This answer not only addresses the question but also indicates the source supporting each second answer fragment through “[1]” and “[2]”. The citation marks in each answer make the statements in the answer traceable to specific source fragments, greatly enhancing the credibility and interpretability of the answer.

[0248] S1305: Generate question-and-answer pairs based on the question, the answer to the question, and the reference tag of the answer. (Based on the question...) For example, its question-answer pairs are ,in, For the question The answer, For the question The context set.

[0249] Use the set of all question-answer pairs in the question dataset as the initial question-answer pair dataset. :

[0250] (15);

[0251] Leveraging existing powerful large language models It has the ability to integrate context sets into answers without additional training, while ensuring a consistent answer style and source annotation.

[0252] The embodiments of this application can ensure that the answer to each question comes from the answer fragment within the context set, and embeds a reference mark for the answer fragment within the answer to ensure the traceability of each important fact.

[0253] In one possible implementation, after constructing the question-answer pair based on the question (step S130), the following steps are also included:

[0254] The question-answer pairs are evaluated for quality. Furthermore, in step S140, the set of all qualified question-answer pairs is compiled into a question-answer pair dataset. .

[0255] Quality assessment can include citation consistency and content standardization assessment (e.g., completeness, length, and semantic reasonableness of the answer). Quality assessment can also include evaluating whether the answer directly and clearly addresses the core intent of the original question. Specifically, another assessment model can be introduced to determine the relevance of the answer to the question and the sufficiency of the answer in the question-answer pair. Since this task is relatively simple, a small-parameter assessment model (such as 4o-mini) can be used for rapid verification.

[0256] This application embodiment performs multi-dimensional quality assessment and screening of question-answer pairs, eliminating low-quality or unreliable data that does not meet preset standards, and finally selecting a high-quality, highly reliable, and formatted reference-based question-answer pair dataset that can be used to train large language models.

[0257] In one possible implementation, the question-answer pairs are quality-assessed, specifically including:

[0258] T1: Determine the validity of the citation of the answer in the question-answer pair.

[0259] Citation validity aims to verify whether the cited information in an answer truly and accurately reflects the content of its claimed source. Specifically, considering both semantic similarity and word overlap, if an answer fails to meet a preset standard with the answer fragment it claims to cite, the citation is deemed "invalid." Invalid answers are considered to contain insufficiently supported content or have errors in the citation markers themselves, and are therefore discarded.

[0260] T2: Determine the length of the answer in the question-answer pair.

[0261] In this step, the character length (CL) of the answers is calculated, and an appropriate length range is set to eliminate answers that are too short or too long. Shorter answers often contain lower information density and fail to fully answer the question, while longer answers contain more redundant noise. Furthermore, outlier answers are marked.

[0262] T3: If the validity and length of the citation are both within the corresponding preset range, the quality assessment result of the question-answer pair is determined to be qualified.

[0263] Understandably, the quality assessment process may also include other quantitative metrics. Only question-answer pairs that pass all quantitative metrics (i.e., meet all set thresholds and rules) will be considered qualified and output to the final question-answer pair dataset. In, that is:

[0264] (16);

[0265] (17);

[0266] in, Indicates obtaining the answer The j Several metrics (such as citation validity, answer length, etc.) , They represent the first j The upper and lower limits of the preset range for each indicator.

[0267] For the aforementioned quantitative indicators (including citation validity and answer length), their distribution on the generated data can be statistically analyzed periodically (e.g., after processing each small batch of data), observing the range of their mean, median, variance, and extreme values. Based on these statistical distributions, the corresponding thresholds can be dynamically adjusted to better suit the characteristics of the current data, rather than relying on a fixed, hard-coded threshold. For example, if it is found that the Jaccard coefficient (related to word overlap, see explanation below) for a certain type of question is generally low, but the semantic similarity is high and the quality of human judgment is acceptable, the Jaccard threshold for that type of question can be appropriately lowered. This dynamic adjustment mechanism helps improve the robustness and accuracy of the screening process.

[0268] In one possible implementation, a manual sampling review step is added to further ensure the quality of question-and-answer pairs. For example, a manual sampling review mechanism can be introduced for question-and-answer pairs that are in a "marginal state" (e.g., some metrics are slightly below the threshold but others are acceptable) or that have been flagged as suspicious by the automatic screening mechanism. Human experts reviewing these samples not only corrects misjudgments but, more importantly, provides valuable feedback for optimizing the automatic screening rules and thresholds.

[0269] This application's embodiments comprehensively evaluate multiple dimensions, including citation validity, answer length, and format conformity, to determine the quality of each question-and-answer pair. This ensures that the final output dataset of question-and-answer pairs with citation tags meets high standards in terms of content accuracy, citation reliability, format conformity, and relevance to the question. Only question-and-answer pairs that meet all preset standards are considered qualified and selected for inclusion in the final question-and-answer pair training dataset. This fundamentally guarantees the traceability of answers and the accuracy of citations, effectively suppressing model illusions and ensuring that the final output dataset is a high-quality and conforming collection of question-and-answer pairs. Such citation-based question-and-answer pairs provide robust supervision signals for training a higher-performing and more reliable large language model, and enable the model to learn to cite sources in its answers while learning the question-and-answer content.

[0270] The answers in a question-and-answer pair and their corresponding source texts should meet the requirements of relevance, support, and accuracy. Relevance requires that the answer be highly relevant to the statements cited in the source text; support requires that the content of the source text truly and accurately supports the statements in the answer, rather than merely being relevant or mentioning certain identical entities. Accuracy requires that the citation mark should point to the most specific and direct text segment that supports the statement, rather than overly broad sections or irrelevant parts.

[0271] Based on the above, in one possible implementation, step T1, determining the citation validity of the answer in the question-answer pair, specifically includes:

[0272] T11: According to the answer The citation marker will be used to indicate the answer. Divide into multiple second-answer segments.

[0273] Specifically, the answer The text is segmented into several second-answer segments based on citation markers and sentence-end markers. For each citation marker in the answer... For example, “[1]”, extract the corresponding second answer fragment. And the first answer fragment that the citation marker points to. Get the answer The set of clause quotations :

[0274] (18);

[0275] T12: Determine the semantic similarity between each second answer fragment and the first answer fragment referenced by the second answer fragment.

[0276] For the set of clause references any element in You can use the second answer fragment. With the first answer fragment Input each component into the same embedding model (e.g., BGE-small) for encoding, and calculate their cosine similarity as their semantic similarity score (SSS). The specific formula is as follows:

[0277] (19);

[0278] Among them, The semantic vector is the output of the embedding model BGE-small. The cosine similarity of a pair of vectors of the same dimension can be calculated, where Represents the dot product of vectors. The L2 norm (Euclidean length) of a vector. The value of is between -1 and 1 (typically 0 to 1 for non-negative vectors), with values ​​closer to 1 indicating greater semantic similarity. Due to the superiority of the BGE-small model, even... and The length differences between them are significant, and the similarity score can reliably reflect the true semantic closeness.

[0279] It reflects the semantic similarity between the first and second answer segments and can capture the deep semantic connection between the first and second answer segments, even if their surface vocabulary is quite different. This reflects, to some extent, the accuracy of the second answer segment. If Exceeding a preset threshold indicates that the answer segment is semantically consistent with the quoted segment. If a pair of A low score indicates that this part of the answer is not semantically related to the source material it claims to cite, and there may be a citation error.

[0280] T13: Determine the lexical overlap rate (LOR) between each second answer fragment and the first answer fragment referenced by the second answer fragment. Specifically, calculate the second answer fragment... With the first answer fragment The n-gram overlap rate between words is also known as the word literal overlap rate.

[0281] In one possible implementation, 3-gram Jaccard coefficients are used to quantify the second answer fragment. With the first answer fragment The phrase co-occurrence rate, i.e., word overlap.

[0282] Specifically, statistical analysis of the second answer segment. With the first answer fragment Each set of three-character (or three-word) sequences is denoted as follows: and And calculate the ratio of their intersection to their union, using the following formula:

[0283] (20);

[0284] Word overlap ratio (LOR) captures the similarity and precision between the second and first answer segments in terms of specific word usage. This is helpful in determining whether it is a "direct quote" or a "close paraphrase," and can also reflect the support to some extent. If the LOR exceeds a preset threshold, it further corroborates that the answer content directly originates from or is closely based on the source segment, reducing the possibility of excessive model paraphrasing or fabrication. If the LOR is very low, it indicates that the second and first answer segments have almost nothing in common in their expression, increasing the suspicion that the answer content is not supported by the source.

[0285] T14: Determine the citation validity score (CVS) of each second answer fragment based on its semantic similarity and word overlap.

[0286] In one possible implementation, normalized weights are used to sum the semantic similarity and word overlap of each second answer fragment to obtain the citation validity of the second answer fragment.

[0287] The citation validity scores are as follows:

[0288] (twenty one);

[0289] in The normalized weight for semantic similarity is taken here. This indicates that, compared to statement support, the system places greater emphasis on whether the content of the second answer fragment is related to the cited first answer fragment. This is an empirical value that performs well in general scenarios. In specific domains, such as legal or academic scenarios that require strict paraphrasing of the original text, it can be appropriately lowered. The value is chosen to increase consideration of word overlap; correspondingly, in popular science or summary scenarios that require high-level summarization and paraphrasing, it can be further improved. .

[0290] T15: Determine the citation validity of the answer based on the citation validity of all second-answer segments in the answer. :

[0291] (twenty two);

[0292] If the answer is valid for reference If the answer falls within its preset range, then the citation of the answer is valid.

[0293] The embodiments of this application reflect the validity of citations through semantic similarity and word overlap. They measure whether the answer's citation of the source text is effective from the perspectives of semantic understanding and word literal construction, and achieve objective evaluation by combining multiple dimensions.

[0294] Based on the above, this application also provides an apparatus for generating a question-answer pair dataset. The apparatus for generating a question-answer pair dataset can be referred to in correspondence with the above-described method for generating a question-answer pair dataset.

[0295] As an example, such as Figure 10 As shown, the question-answer pair dataset generation device provided in this application includes a segmentation module 1010, a context set acquisition module 1020, a question-answer pair acquisition module 1030, and a dataset acquisition module 1040.

[0296] The segmentation module 1010 is used to segment each answer data source in the answer dataset to obtain multiple first answer fragments, each of which is configured with source metadata.

[0297] The context set acquisition module 1020 is used to match at least one first answer fragment for each question in the question dataset and construct a context set for generating answers to the questions.

[0298] The question-answer pair acquisition module 1030 is used to generate an answer based on the question and the context set of the question for any question in the question dataset, and to construct a question-answer pair based on the question and the answer. The question-answer pair includes the question, the answer to the question, and the reference tag of the answer.

[0299] The dataset acquisition module 1040 is used to construct a question-answer pair dataset based on the set of question-answer pairs corresponding to all questions.

[0300] In this application embodiment, source metadata is attached to each answer fragment in the answer dataset, so that the answer to the question pair is configured with clear source metadata, so that the answer can be traced back to a specific data source, and the answer output by the large language model has credibility and accuracy.

[0301] In one possible implementation, the question-answer pair dataset generation device further includes a quality evaluation module for evaluating the quality of the question-answer pairs. In this embodiment, the dataset acquisition module is used to collect the set of all qualified question-answer pairs as the question-answer pair dataset.

[0302] This application embodiment performs quality assessment and screening of question-answer pairs, eliminating low-quality or unreliable data that does not meet preset standards, and finally selecting a high-quality, highly reliable, and formatted reference-based question-answer pair dataset that can be used to train large language models.

[0303] In one possible implementation, the device for generating the question-answering pair dataset further includes an index vector library building module, which is used for:

[0304] Convert each first answer fragment into a first semantic vector;

[0305] An index vector library is built based on the first semantic vector and source metadata of all first answer fragments.

[0306] This application's embodiments, through the embedding model and efficient vector database, ensure high-quality semantic similarity search in a large-scale fragment library with a sub-second response speed. This provides the conditions for matching question and answer fragments, lays a solid foundation for subsequent real-time and accurate question matching and large-scale question-answer pair generation, and guarantees scalability.

[0307] Based on the above, this application also provides a question-answering model, which is trained on a question-answering pair dataset obtained by any of the above-mentioned question-answering pair dataset generation methods.

[0308] Figure 11 This is a schematic diagram of the structure of the electronic device provided in this application, such as... Figure 11 As shown, the electronic device may include a processor 1110, a communications interface 1120, a memory 1130, and a communication bus 1140, wherein the processor 1110, the communications interface 1120, and the memory 1130 communicate with each other via the communication bus 1140. The processor 1110 can call logical instructions in the memory 1130 to execute a method for generating a question-answer pair dataset. This method includes: segmenting each answer data source in the answer dataset to obtain multiple first answer fragments, each first answer fragment being configured with source metadata; matching at least one first answer fragment for each question in the question dataset to construct a context set for generating answers to the questions; for any question in the question dataset, generating an answer based on the question and the question's context set; constructing a question-answer pair based on the question and the answer, the question-answer pair including the question, the answer to the question, and a reference tag for the answer; and constructing a question-answer pair dataset based on the set of question-answer pairs corresponding to all questions.

[0309] Furthermore, the logical instructions in the aforementioned memory 1130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0310] On the other hand, this application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer is able to execute the question-answer pair dataset generation method provided in the above embodiments. The method includes: segmenting each answer data source in the answer dataset to obtain multiple first answer fragments, each first answer fragment being configured with source metadata; matching at least one first answer fragment for each question in the question dataset to construct a context set for generating answers to the questions; for any question in the question dataset, generating an answer based on the question and the question's context set, constructing a question-answer pair based on the question and the answer, the question-answer pair including the question, the answer to the question, and a reference tag for the answer; and constructing a question-answer pair dataset based on the set of question-answer pairs corresponding to all questions.

[0311] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program is implemented to perform the question-answer pair dataset generation method provided in the above embodiments. The method includes: segmenting each answer data source in the answer dataset to obtain multiple first answer fragments, each first answer fragment being configured with source metadata; matching at least one first answer fragment for each question in the question dataset to construct a context set for generating answers to the questions; for any question in the question dataset, generating an answer based on the question and the question's context set; constructing a question-answer pair based on the question and the answer, the question-answer pair including the question, the answer to the question, and a reference tag for the answer; and constructing a question-answer pair dataset based on the set of question-answer pairs corresponding to all questions.

[0312] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0313] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0314] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for generating a question and answer pair dataset, the method comprising: The method comprises the following steps: segmenting each answer data source in an answer data set to obtain a plurality of first answer segments, each of which is configured with source metadata; matching at least one of the first answer segments with each question in a question data set to construct a context set for generating an answer to the question; generating an answer based on the question and the context set of the question for any question in the question data set, and constructing a question and answer pair based on the question and the answer, which comprises the question, the answer to the question, and a reference mark of the answer; constructing a question and answer pair data set according to the set of question and answer pairs corresponding to all the questions; each of the questions in the question data set is configured with at least one answer source identifier, each of which points to a first target answer data source in the answer data set; wherein the question data set is obtained, specifically comprising: extracting first key information of each source question in a question source data set, the first key information being information related to the answer data source, the first key information comprising the main content involved in the source question or the basic information of the answer data source, the main content comprising plot, character, setting, concept, and theme; matching the first key information with second key information of each answer data source in the answer data set; if there is second key information matching the first key information, the source question is taken as a question in the question data set, and the source metadata of the answer data source to which the second key information belongs is taken as the answer source identifier of the source question.

2. The method of Claim 1, wherein, After the question and answer pair based on the question is constructed, the method further comprises: evaluating the quality of the question and answer pair; and taking the set of all qualified question and answer pairs as the question and answer pair data set.

3. The method of Claim 1, wherein, After the plurality of first answer segments are obtained by segmenting each answer data source in the answer data set, the method further comprises: converting each of the first answer segments into a first semantic vector; constructing an index vector library based on the first semantic vectors of all the first answer segments and the source metadata.

4. The method of Claim 3, wherein, The method of matching at least one of the first answer segments with each question in the question data set to construct a context set for generating an answer to the question specifically comprises: converting the question into a second semantic vector; determining the search range of the question in the index vector library based on the answer source identifier of the question; retrieving the first predetermined number of candidate answer segments with the highest matching degree with the second semantic vector in the search range to construct a candidate answer segment set; constructing a context set for generating an answer to the question based on the candidate answer segment set.

5. The method of Claim 4, wherein, After the candidate answer segment set is obtained, the method further comprises: selecting a preferred answer segment set from the candidate answer segment set based on semantic similarity; and constructing a context set for generating an answer to the question based on the preferred answer segment set.

6. The method of Claim 5, wherein, The method of constructing a context set for generating an answer to the question based on the preferred answer segment set specifically comprises: configuring a unique reference mark for each preferred answer segment in the set of preferred answer segments; combining each of the preferred answer segments and the reference mark of the preferred answer segment into a reference pair; combining the set of reference pairs of all the preferred answer segments as the set of contexts of the question.

7. The method of Claim 1, wherein, In extracting the first key information of the source question, specifically comprising: determining the relevance of the main content of the source question to the second target answer data source; if the relevance is higher than a second threshold, taking the basic information of the second target answer data source as part of the first key information of the source question.

8. The method of Claim 1 or 7, wherein, The obtaining of the question data set further comprises: when the asking distance corresponding to any answer data source in the question data set is greater than a third threshold, expanding the number of questions corresponding to the any answer data source, and combining the set of all expanded questions and the set of all source questions into the question data set.

9. The method of Claim 8, wherein, The expanding of the number of questions corresponding to the any answer data source specifically comprises: determining the expansion amount of the any answer data source; extracting at least one seed segment from all first answer segments of the any answer data source; generating at least one first expanded question for each of the seed segments.

10. The method of Claim 9, wherein, After extracting at least one seed segment, further comprising: extracting a set of reference questions from the source questions corresponding to the any answer data source; generating at least one first expanded question for each of the seed segments with the set of reference questions as a reference.

11. The method of Claim 9, wherein, After generating at least one first expanded question for each of the seed segments, further comprising: performing quality screening on each of the first expanded questions to obtain at least one second expanded question; adding source metadata and a source to each of the second expanded questions, the source indicating that the second expanded question is obtained through an expansion approach.

12. The method of Claim 1, wherein, Based on the question and the set of contexts of the question, generating an answer, and based on the question and the answer, constructing a question and answer pair based on the question, specifically comprising: constructing a prompt word based on the question and the set of contexts of the question; adding a constraint instruction in the prompt word; constructing an output format; inputting the prompt word and the output format into a large language model to obtain an answer to the question generated by the large language model, the answer being configured with a reference mark; generating a question and answer pair based on the question, the answer to the question, and the reference mark of the answer.

13. The method of Claim 5, wherein, Based on semantic similarity, filtering a set of preferred answer segments from the set of candidate answer segments, specifically comprising: determining a first similarity between each of the candidate answer segments and the second semantic vector; determining a second similarity between each of the candidate answer segments and any other candidate answer segment in the set of candidate answer segments; obtaining the weighted sum of the first similarity and all the second similarities corresponding to each of the candidate answer segments to determine the score of the candidate answer segment; taking the set of the second preset number of candidate answer segments with the highest score as the set of preferred answer segments.

14. The method of Claim 2, wherein, The quality evaluation of the question and answer pair specifically comprises: determining the reference validity of the answer in the question and answer pair; Determine the length of the answer in the question-answer pair; If both the validity of the citation and the length are within the corresponding preset range, the quality assessment result of the question-answer pair is determined to be qualified.

15. The method of Claim 14, wherein, Determining the validity of the citation of the answers in the question-answer pair specifically includes: Based on the reference markers of the answer, the answer is divided into multiple second answer segments; Determine the semantic similarity between each second answer fragment and the first answer fragment referenced by the second answer fragment; Determine the literal overlap between each second answer fragment and the first answer fragment referenced by the second answer fragment; Using normalized weights, the semantic similarity and word overlap of each second answer segment are weighted and summed to obtain the citation validity of the second answer segment; The citation validity of the answer is determined based on the citation validity of all second answer fragments in the answer.

16. An apparatus for generating a question-answer pair dataset, the apparatus comprising: It includes a segmentation module, a context set acquisition module, a question-answer pair acquisition module, and a dataset acquisition module; The segmentation module is used to segment each answer data source in the answer dataset to obtain multiple first answer fragments, each of which is configured with source metadata; The context set acquisition module is used to match at least one of the first answer fragments for each question in the question dataset, and construct a context set for generating the answer to the question; The question-answer pair acquisition module is used to generate an answer based on the question and the context set of the question for any question in the question dataset, and to construct a question-answer pair based on the question and the answer. The question-answer pair includes the question, the answer to the question, and the reference tag of the answer. The dataset acquisition module is used to construct a question-answer pair dataset based on the set of question-answer pairs corresponding to all the questions; Each question in the question dataset is configured with at least one answer source identifier, and each answer source identifier points to a first target answer data source in the answer dataset. The question-and-answer pair dataset generation device is further used to obtain the question dataset, specifically including: Extract the first key information of each source question in the source question dataset. The first key information is information related to the answer data source. The first key information includes the main content of the source question or the basic information of the answer data source. The main content includes plot, characters, settings, concepts, and themes. The first key information is matched with the second key information of each answer data source in the answer dataset; If a second key information exists that matches the first key information, then the source question is taken as a question in the question dataset, and the source metadata corresponding to the answer data source to which the second key information belongs is taken as the answer source identifier of the source question.

17. A question and answer model, the method comprising: The question-answering model is trained on a question-answering pair dataset obtained by the question-answering pair dataset generation method according to any one of claims 1 to 15.

18. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, When the processor executes the computer program, it implements the question-answer pair dataset generation method as described in any one of claims 1 to 15. 19.A non-transitory computer-readable storage medium having stored thereon a computer program, wherein, When the computer program is executed by a processor, it implements the method for generating a question-answer pair dataset as described in any one of claims 1 to 15.

20. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method for generating a question-answer pair dataset as described in any one of claims 1 to 15.

Images